* HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
@ 2017-09-13 15:20 Denys Fedoryshchenko
2017-09-13 15:34 ` Denys Fedoryshchenko
2017-09-13 15:51 ` HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Eric Dumazet
0 siblings, 2 replies; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 15:20 UTC (permalink / raw)
To: Linux Kernel Network Developers
Hi,
I noticed after increasing bandwidth over some amount HTB started to
throttle classes it should not throttle.
Also estimated rate in htb totally wrong, while byte counters is
correct.
Is there any overflow or something?
X520 card (but XL710 same)
br1 8000.90e2ba86c38c no eth3.1777
eth3.777
br2 8000.90e2ba86c38d no eth3.360
eth3.361
Inbound traffic is coming over one vlan, leaving another vlan.
Shaper is just bunch of classes and u32 filters, with few fw filters.
qdisc is pie
I put totally high values to not reach them, tried to change
quantum/burst/cburst but... stats below.
First, "root" class is 1:1 showing rate 18086Mbit, which is physically
impossible.
Class 1:111 showing 5355Mbit, while real traffic is ~1.5Gbit
shaper /etc # tc -s -d class show dev eth3.777 classid 1:111;sleep 5;tc
-s -d class show dev eth3.777 classid 1:111
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 6487632263 bytes 5235525 pkt (dropped 0, overlimits 0 requeues 0)
rate 5529Mbit 557534pps backlog 0b 0p requeues 0
lended: 2423323 borrowed: 0 giants: 0
tokens: 124 ctokens: -1
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 7438601731 bytes 6003811 pkt (dropped 0, overlimits 0 requeues 0)
rate 5631Mbit 568214pps backlog 36624b 8p requeues 0
lended: 2772486 borrowed: 0 giants: 0
tokens: 124 ctokens: -1
(7438601731-6487632263)/5*8 = 1.521.551.148
And most important some classes suffering, while they should not (not
reaching limits)
class htb 1:95 parent 1:1 leaf 95: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 13556762059 bytes 17474559 pkt (dropped 16017, overlimits 0
requeues 0)
rate 2524Mbit 414197pps backlog 31969245b 34513p requeues 0
lended: 13995723 borrowed: 0 giants: 0
tokens: 111 ctokens: -2
Full classes stats:
class htb 1:100 parent 1:1 leaf 100: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 116 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
rate 8bit 0pps backlog 0b 0p requeues 0
lended: 2 borrowed: 0 giants: 0
tokens: 124 ctokens: -1
class htb 1:120 parent 1:1 leaf 120: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 531230043 bytes 782130 pkt (dropped 0, overlimits 0 requeues 0)
rate 132274Kbit 25240pps backlog 0b 0p requeues 0
lended: 540693 borrowed: 0 giants: 0
tokens: 109 ctokens: -2
class htb 1:50 parent 1:1 leaf 50: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 773472109 bytes 587335 pkt (dropped 0, overlimits 0 requeues 0)
rate 215929Kbit 20503pps backlog 0b 0p requeues 0
lended: 216614 borrowed: 0 giants: 0
tokens: 91 ctokens: -4
class htb 1:70 parent 1:1 leaf 70: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 1574768 bytes 6194 pkt (dropped 0, overlimits 0 requeues 0)
rate 406272bit 214pps backlog 0b 0p requeues 0
lended: 5758 borrowed: 0 giants: 0
tokens: 101 ctokens: -3
class htb 1:90 parent 1:1 leaf 90: prio 0 quantum 50000 rate 1Kbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 3206 bytes 53 pkt (dropped 0, overlimits 0 requeues 0)
rate 848bit 1pps backlog 0b 0p requeues 0
lended: 53 borrowed: 0 giants: 0
class htb 1:110 parent 1:1 leaf 110: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 17205952113 bytes 12926008 pkt (dropped 239, overlimits 0 requeues
0)
rate 4433Mbit 416825pps backlog 5847785b 2394p requeues 0
lended: 7021696 borrowed: 0 giants: 0
tokens: 91 ctokens: -4
class htb 1:45 root leaf 45: prio 0 quantum 50000 rate 80Mbit ceil
80Mbit linklayer ethernet burst 10000b/1 mpu 0b cburst 10000b/1 mpu 0b
level 0
Sent 2586 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
rate 456bit 1pps backlog 0b 0p requeues 0
lended: 45 borrowed: 0 giants: 0
tokens: 15540 ctokens: 15540
class htb 1:1 root rate 100Gbit ceil 100Gbit linklayer ethernet burst
0b/1 mpu 0b cburst 0b/1 mpu 0b level 7
Sent 72277215121 bytes 72693012 pkt (dropped 0, overlimits 0 requeues
0)
rate 18086Mbit 2304729pps backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: -4 ctokens: -4
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 21977384237 bytes 17697345 pkt (dropped 391, overlimits 0 requeues
0)
rate 5355Mbit 543216pps backlog 41746345b 14743p requeues 0
lended: 8837667 borrowed: 0 giants: 0
tokens: 124 ctokens: -1
class htb 1:200 parent 1:1 leaf 200: prio 0 quantum 50000 rate 10Gbit
ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 9209112058 bytes 10941655 pkt (dropped 5396413, overlimits 0
requeues 0)
rate 2142Mbit 326938pps backlog 20798724b 25717p requeues 0
lended: 9926866 borrowed: 0 giants: 0
tokens: 108 ctokens: -2
class htb 1:20 parent 1:1 leaf 20: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 1504918577 bytes 1170037 pkt (dropped 0, overlimits 0 requeues 0)
rate 413892Kbit 40322pps backlog 0b 0p requeues 0
lended: 497069 borrowed: 0 giants: 0
tokens: 92 ctokens: -4
class htb 1:40 parent 1:1 leaf 40: prio 0 quantum 50000 rate 1Gbit ceil
1Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 10000b/1 mpu 0b
level 0
Sent 8638241052 bytes 9526035 pkt (dropped 0, overlimits 0 requeues 0)
rate 2442Mbit 333804pps backlog 0b 0p requeues 0
lended: 6947694 borrowed: 0 giants: 0
tokens: 1064 ctokens: 1064
class htb 1:60 parent 1:1 leaf 60: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 3124889750 bytes 6972789 pkt (dropped 5833, overlimits 0 requeues
0)
rate 824981Kbit 233296pps backlog 616b 3p requeues 0
lended: 6701559 borrowed: 0 giants: 0
tokens: 122 ctokens: -1
class htb 1:25 parent 1:1 leaf 25: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 3275 bytes 57 pkt (dropped 0, overlimits 0 requeues 0)
rate 504bit 1pps backlog 0b 0p requeues 0
lended: 57 borrowed: 0 giants: 0
tokens: 123 ctokens: -1
class htb 1:80 parent 1:1 leaf 80: prio 0 quantum 50000 rate 10Gbit ceil
100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu 0b
level 0
Sent 2598 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
rate 672bit 1pps backlog 0b 0p requeues 0
lended: 45 borrowed: 0 giants: 0
tokens: 124 ctokens: -1
tc -s -d qdisc show dev eth3.777
qdisc htb 1: root refcnt 2 r2q 10 default 200 direct_packets_stat 6858
ver 3.17 direct_qlen 1000
Sent 185271955178 bytes 186767377 pkt (dropped 14248387, overlimits 0
requeues 0)
backlog 62369772b 68728p requeues 0
qdisc pie 120: parent 1:120 limit 40000p target 200.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 1532094051 bytes 2108158 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 1424069 overlimit 0 dropped 0 maxq 96 ecn_mark 0
qdisc pie 95: parent 1:95 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 24359099839 bytes 31531160 pkt (dropped 36527, overlimits 0
requeues 0)
backlog 27116831b 28680p requeues 0
prob 0.000000 delay 82019us avg_dq_rate 96252441
pkts_in 25249681 overlimit 2107 dropped 36527 maxq 39999 ecn_mark 0
qdisc pie 80: parent 1:80 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 19695 bytes 317 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 317 overlimit 0 dropped 0 maxq 1 ecn_mark 0
qdisc pie 111: parent 1:111 limit 40000p target 200.0ms tupdate 300.0ms
alpha 2 beta 20
Sent 57513872600 bytes 46387754 pkt (dropped 0, overlimits 0 requeues
0)
backlog 126218b 49p requeues 0
prob 0.000000 delay 227us avg_dq_rate 184143066
qdisc pie 100: parent 1:100 limit 40000p target 200.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 494 bytes 9 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 9 overlimit 0 dropped 0 maxq 0 ecn_mark 0
qdisc pie 60: parent 1:60 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 8749993535 bytes 18718111 pkt (dropped 0, overlimits 0 requeues 0)
backlog 7875b 14p requeues 0
prob 0.000000 delay 98us avg_dq_rate 0
pkts_in 17978754 overlimit 0 dropped 0 maxq 281 ecn_mark 0
qdisc pie 25: parent 1:25 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 9412 bytes 135 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 135 overlimit 0 dropped 0 maxq 0 ecn_mark 0
qdisc pie 200: parent 1:200 limit 40000p target 50.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 22224392951 bytes 26861660 pkt (dropped 14211817, overlimits 0
requeues 0)
backlog 35124441b 39999p requeues 0
prob 0.000071 delay 23747us avg_dq_rate 66040039
pkts_in 25266532 overlimit 14160730 dropped 14211817 maxq 39999 ecn_mark
0
qdisc pie 90: parent 1:90 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 7396 bytes 134 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 134 overlimit 0 dropped 0 maxq 0 ecn_mark 0
qdisc pie 40: parent 1:40 limit 40000p target 200.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 21316494470 bytes 23786396 pkt (dropped 0, overlimits 0 requeues
0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 2085892079
pkts_in 17465291 overlimit 0 dropped 0 maxq 184 ecn_mark 0
qdisc pie 110: parent 1:110 limit 40000p target 200.0ms tupdate 300.0ms
alpha 2 beta 20
Sent 44081304635 bytes 33108042 pkt (dropped 0, overlimits 0 requeues
0)
backlog 0b 0p requeues 0
prob 0.000000 delay 47us avg_dq_rate 201721191
pkts_in 17602600 overlimit 0 dropped 0 maxq 157 ecn_mark 0
qdisc pie 70: parent 1:70 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 4992265 bytes 16959 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 15448 overlimit 0 dropped 0 maxq 3 ecn_mark 0
qdisc pie 20: parent 1:20 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20
Sent 3475884019 bytes 2717342 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 1186311 overlimit 0 dropped 0 maxq 41 ecn_mark 0
qdisc pie 50: parent 1:50 limit 40000p target 250.0ms tupdate 30.0ms
alpha 2 beta 20 ecn
Sent 1994354891 bytes 1511636 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
prob 0.000000 delay 0us avg_dq_rate 0
pkts_in 561648 overlimit 0 dropped 0 maxq 27 ecn_mark 0
More info about setup:
shaper /etc # ethtool -k eth3
Features for eth3:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: on [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
shaper /etc # ethtool -k eth3.777
Features for eth3.777:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: on
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [requested on]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
shaper /etc # ethtool -i eth3
driver: ixgbe
version: 5.0.0-k
firmware-version: 0x2b2c0001
expansion-rom-version:
bus-info: 0000:06:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
mpstat -P ALL 10
Linux 4.12.9-build-0132 (shaper) 09/13/17 _x86_64_ (32 CPU)
18:15:55 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
18:16:05 all 0.00 0.00 0.00 0.00 0.00 0.63 0.00
0.00 99.37
18:16:05 0 0.00 0.00 0.13 0.00 0.00 0.13 0.00
0.00 99.73
18:16:05 1 0.00 0.00 0.00 0.00 0.00 1.34 0.00
0.00 98.66
18:16:05 2 0.00 0.00 0.00 0.00 0.00 1.32 0.00
0.00 98.68
18:16:05 3 0.00 0.00 0.00 0.00 0.00 1.81 0.00
0.00 98.19
18:16:05 4 0.00 0.00 0.00 0.00 0.00 1.78 0.00
0.00 98.22
18:16:05 5 0.00 0.00 0.00 0.00 0.00 1.77 0.00
0.00 98.23
18:16:05 6 0.00 0.00 0.00 0.00 0.00 1.77 0.00
0.00 98.23
18:16:05 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 8 0.00 0.00 0.00 0.00 0.00 0.15 0.00
0.00 99.85
18:16:05 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 10 0.00 0.00 0.00 0.00 0.00 0.15 0.00
0.00 99.85
18:16:05 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 12 0.00 0.00 0.00 0.00 0.00 0.48 0.00
0.00 99.52
18:16:05 13 0.00 0.00 0.00 0.00 0.00 0.59 0.00
0.00 99.41
18:16:05 14 0.00 0.00 0.00 0.00 0.00 1.30 0.00
0.00 98.70
18:16:05 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 16 0.00 0.00 0.00 0.00 0.00 0.12 0.00
0.00 99.88
18:16:05 17 0.00 0.00 0.00 0.00 0.00 0.82 0.00
0.00 99.18
18:16:05 18 0.00 0.00 0.00 0.00 0.00 0.71 0.00
0.00 99.29
18:16:05 19 0.00 0.00 0.00 0.00 0.00 0.71 0.00
0.00 99.29
18:16:05 20 0.00 0.00 0.00 0.00 0.00 0.72 0.00
0.00 99.28
18:16:05 21 0.00 0.00 0.00 0.00 0.00 0.59 0.00
0.00 99.41
18:16:05 22 0.00 0.00 0.00 0.00 0.00 1.30 0.00
0.00 98.70
18:16:05 23 0.00 0.00 0.00 0.00 0.00 1.18 0.00
0.00 98.82
18:16:05 24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 25 0.00 0.00 0.00 0.00 0.00 0.84 0.00
0.00 99.16
18:16:05 26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 27 0.00 0.00 0.00 0.00 0.00 0.85 0.00
0.00 99.15
18:16:05 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 29 0.00 0.00 0.12 0.00 0.00 0.12 0.00
0.00 99.76
18:16:05 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
18:16:05 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 15:20 HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Denys Fedoryshchenko
@ 2017-09-13 15:34 ` Denys Fedoryshchenko
2017-09-13 16:16 ` Eric Dumazet
2017-09-13 15:51 ` HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Eric Dumazet
1 sibling, 1 reply; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 15:34 UTC (permalink / raw)
To: Linux Kernel Network Developers
Well, probably i am answering my own question, removing estimator from
classes seems drastically improve situation.
It seems estimator has some issues that cause shaper to behave
incorrectly (throttling traffic while it should not).
But i guess thats a bug?
As i was not able to predict such bottleneck by CPU load measurements.
On 2017-09-13 18:20, Denys Fedoryshchenko wrote:
> Hi,
>
> I noticed after increasing bandwidth over some amount HTB started to
> throttle classes it should not throttle.
> Also estimated rate in htb totally wrong, while byte counters is
> correct.
>
> Is there any overflow or something?
>
> X520 card (but XL710 same)
> br1 8000.90e2ba86c38c no eth3.1777
> eth3.777
> br2 8000.90e2ba86c38d no eth3.360
> eth3.361
>
> Inbound traffic is coming over one vlan, leaving another vlan.
> Shaper is just bunch of classes and u32 filters, with few fw filters.
> qdisc is pie
>
> I put totally high values to not reach them, tried to change
> quantum/burst/cburst but... stats below.
>
> First, "root" class is 1:1 showing rate 18086Mbit, which is
> physically impossible.
>
> Class 1:111 showing 5355Mbit, while real traffic is ~1.5Gbit
>
> shaper /etc # tc -s -d class show dev eth3.777 classid 1:111;sleep
> 5;tc -s -d class show dev eth3.777 classid 1:111
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 6487632263 bytes 5235525 pkt (dropped 0, overlimits 0 requeues 0)
> rate 5529Mbit 557534pps backlog 0b 0p requeues 0
> lended: 2423323 borrowed: 0 giants: 0
> tokens: 124 ctokens: -1
>
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 7438601731 bytes 6003811 pkt (dropped 0, overlimits 0 requeues 0)
> rate 5631Mbit 568214pps backlog 36624b 8p requeues 0
> lended: 2772486 borrowed: 0 giants: 0
> tokens: 124 ctokens: -1
>
> (7438601731-6487632263)/5*8 = 1.521.551.148
>
> And most important some classes suffering, while they should not (not
> reaching limits)
> class htb 1:95 parent 1:1 leaf 95: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 13556762059 bytes 17474559 pkt (dropped 16017, overlimits 0
> requeues 0)
> rate 2524Mbit 414197pps backlog 31969245b 34513p requeues 0
> lended: 13995723 borrowed: 0 giants: 0
> tokens: 111 ctokens: -2
>
>
>
>
>
> Full classes stats:
>
> class htb 1:100 parent 1:1 leaf 100: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 116 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> rate 8bit 0pps backlog 0b 0p requeues 0
> lended: 2 borrowed: 0 giants: 0
> tokens: 124 ctokens: -1
>
> class htb 1:120 parent 1:1 leaf 120: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 531230043 bytes 782130 pkt (dropped 0, overlimits 0 requeues 0)
> rate 132274Kbit 25240pps backlog 0b 0p requeues 0
> lended: 540693 borrowed: 0 giants: 0
> tokens: 109 ctokens: -2
>
> class htb 1:50 parent 1:1 leaf 50: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 773472109 bytes 587335 pkt (dropped 0, overlimits 0 requeues 0)
> rate 215929Kbit 20503pps backlog 0b 0p requeues 0
> lended: 216614 borrowed: 0 giants: 0
> tokens: 91 ctokens: -4
>
> class htb 1:70 parent 1:1 leaf 70: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 1574768 bytes 6194 pkt (dropped 0, overlimits 0 requeues 0)
> rate 406272bit 214pps backlog 0b 0p requeues 0
> lended: 5758 borrowed: 0 giants: 0
> tokens: 101 ctokens: -3
>
> class htb 1:90 parent 1:1 leaf 90: prio 0 quantum 50000 rate 1Kbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 3206 bytes 53 pkt (dropped 0, overlimits 0 requeues 0)
> rate 848bit 1pps backlog 0b 0p requeues 0
> lended: 53 borrowed: 0 giants: 0
>
> class htb 1:110 parent 1:1 leaf 110: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 17205952113 bytes 12926008 pkt (dropped 239, overlimits 0
> requeues 0)
> rate 4433Mbit 416825pps backlog 5847785b 2394p requeues 0
> lended: 7021696 borrowed: 0 giants: 0
> tokens: 91 ctokens: -4
>
> class htb 1:45 root leaf 45: prio 0 quantum 50000 rate 80Mbit ceil
> 80Mbit linklayer ethernet burst 10000b/1 mpu 0b cburst 10000b/1 mpu 0b
> level 0
> Sent 2586 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
> rate 456bit 1pps backlog 0b 0p requeues 0
> lended: 45 borrowed: 0 giants: 0
> tokens: 15540 ctokens: 15540
>
> class htb 1:1 root rate 100Gbit ceil 100Gbit linklayer ethernet burst
> 0b/1 mpu 0b cburst 0b/1 mpu 0b level 7
> Sent 72277215121 bytes 72693012 pkt (dropped 0, overlimits 0 requeues
> 0)
> rate 18086Mbit 2304729pps backlog 0b 0p requeues 0
> lended: 0 borrowed: 0 giants: 0
> tokens: -4 ctokens: -4
>
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 21977384237 bytes 17697345 pkt (dropped 391, overlimits 0
> requeues 0)
> rate 5355Mbit 543216pps backlog 41746345b 14743p requeues 0
> lended: 8837667 borrowed: 0 giants: 0
> tokens: 124 ctokens: -1
>
> class htb 1:200 parent 1:1 leaf 200: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 9209112058 bytes 10941655 pkt (dropped 5396413, overlimits 0
> requeues 0)
> rate 2142Mbit 326938pps backlog 20798724b 25717p requeues 0
> lended: 9926866 borrowed: 0 giants: 0
> tokens: 108 ctokens: -2
>
> class htb 1:20 parent 1:1 leaf 20: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 1504918577 bytes 1170037 pkt (dropped 0, overlimits 0 requeues 0)
> rate 413892Kbit 40322pps backlog 0b 0p requeues 0
> lended: 497069 borrowed: 0 giants: 0
> tokens: 92 ctokens: -4
>
> class htb 1:40 parent 1:1 leaf 40: prio 0 quantum 50000 rate 1Gbit
> ceil 1Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 10000b/1
> mpu 0b level 0
> Sent 8638241052 bytes 9526035 pkt (dropped 0, overlimits 0 requeues 0)
> rate 2442Mbit 333804pps backlog 0b 0p requeues 0
> lended: 6947694 borrowed: 0 giants: 0
> tokens: 1064 ctokens: 1064
>
> class htb 1:60 parent 1:1 leaf 60: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 3124889750 bytes 6972789 pkt (dropped 5833, overlimits 0 requeues
> 0)
> rate 824981Kbit 233296pps backlog 616b 3p requeues 0
> lended: 6701559 borrowed: 0 giants: 0
> tokens: 122 ctokens: -1
>
> class htb 1:25 parent 1:1 leaf 25: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 3275 bytes 57 pkt (dropped 0, overlimits 0 requeues 0)
> rate 504bit 1pps backlog 0b 0p requeues 0
> lended: 57 borrowed: 0 giants: 0
> tokens: 123 ctokens: -1
>
> class htb 1:80 parent 1:1 leaf 80: prio 0 quantum 50000 rate 10Gbit
> ceil 100Gbit linklayer ethernet burst 10000b/1 mpu 0b cburst 0b/1 mpu
> 0b level 0
> Sent 2598 bytes 45 pkt (dropped 0, overlimits 0 requeues 0)
> rate 672bit 1pps backlog 0b 0p requeues 0
> lended: 45 borrowed: 0 giants: 0
> tokens: 124 ctokens: -1
>
> tc -s -d qdisc show dev eth3.777
> qdisc htb 1: root refcnt 2 r2q 10 default 200 direct_packets_stat 6858
> ver 3.17 direct_qlen 1000
> Sent 185271955178 bytes 186767377 pkt (dropped 14248387, overlimits 0
> requeues 0)
> backlog 62369772b 68728p requeues 0
> qdisc pie 120: parent 1:120 limit 40000p target 200.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 1532094051 bytes 2108158 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 1424069 overlimit 0 dropped 0 maxq 96 ecn_mark 0
>
> qdisc pie 95: parent 1:95 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 24359099839 bytes 31531160 pkt (dropped 36527, overlimits 0
> requeues 0)
> backlog 27116831b 28680p requeues 0
> prob 0.000000 delay 82019us avg_dq_rate 96252441
> pkts_in 25249681 overlimit 2107 dropped 36527 maxq 39999 ecn_mark 0
>
> qdisc pie 80: parent 1:80 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 19695 bytes 317 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 317 overlimit 0 dropped 0 maxq 1 ecn_mark 0
>
> qdisc pie 111: parent 1:111 limit 40000p target 200.0ms tupdate
> 300.0ms alpha 2 beta 20
> Sent 57513872600 bytes 46387754 pkt (dropped 0, overlimits 0 requeues
> 0)
> backlog 126218b 49p requeues 0
> prob 0.000000 delay 227us avg_dq_rate 184143066
>
> qdisc pie 100: parent 1:100 limit 40000p target 200.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 494 bytes 9 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 9 overlimit 0 dropped 0 maxq 0 ecn_mark 0
>
> qdisc pie 60: parent 1:60 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 8749993535 bytes 18718111 pkt (dropped 0, overlimits 0 requeues
> 0)
> backlog 7875b 14p requeues 0
> prob 0.000000 delay 98us avg_dq_rate 0
> pkts_in 17978754 overlimit 0 dropped 0 maxq 281 ecn_mark 0
>
> qdisc pie 25: parent 1:25 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 9412 bytes 135 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 135 overlimit 0 dropped 0 maxq 0 ecn_mark 0
>
> qdisc pie 200: parent 1:200 limit 40000p target 50.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 22224392951 bytes 26861660 pkt (dropped 14211817, overlimits 0
> requeues 0)
> backlog 35124441b 39999p requeues 0
> prob 0.000071 delay 23747us avg_dq_rate 66040039
> pkts_in 25266532 overlimit 14160730 dropped 14211817 maxq 39999
> ecn_mark
> 0
>
> qdisc pie 90: parent 1:90 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 7396 bytes 134 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 134 overlimit 0 dropped 0 maxq 0 ecn_mark 0
>
> qdisc pie 40: parent 1:40 limit 40000p target 200.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 21316494470 bytes 23786396 pkt (dropped 0, overlimits 0 requeues
> 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 2085892079
> pkts_in 17465291 overlimit 0 dropped 0 maxq 184 ecn_mark 0
>
> qdisc pie 110: parent 1:110 limit 40000p target 200.0ms tupdate
> 300.0ms alpha 2 beta 20
> Sent 44081304635 bytes 33108042 pkt (dropped 0, overlimits 0 requeues
> 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 47us avg_dq_rate 201721191
> pkts_in 17602600 overlimit 0 dropped 0 maxq 157 ecn_mark 0
>
> qdisc pie 70: parent 1:70 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 4992265 bytes 16959 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 15448 overlimit 0 dropped 0 maxq 3 ecn_mark 0
>
> qdisc pie 20: parent 1:20 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20
> Sent 3475884019 bytes 2717342 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 1186311 overlimit 0 dropped 0 maxq 41 ecn_mark 0
>
> qdisc pie 50: parent 1:50 limit 40000p target 250.0ms tupdate 30.0ms
> alpha 2 beta 20 ecn
> Sent 1994354891 bytes 1511636 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> prob 0.000000 delay 0us avg_dq_rate 0
> pkts_in 561648 overlimit 0 dropped 0 maxq 27 ecn_mark 0
>
>
>
> More info about setup:
> shaper /etc # ethtool -k eth3
> Features for eth3:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: on [fixed]
> tx-checksum-sctp: on
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: off
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: on
> highdma: on [fixed]
> rx-vlan-filter: on
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: on [fixed]
> tx-gre-segmentation: on
> tx-gre-csum-segmentation: on
> tx-ipxip4-segmentation: on
> tx-ipxip6-segmentation: on
> tx-udp_tnl-segmentation: on
> tx-udp_tnl-csum-segmentation: on
> rx-all: off
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off
> hw-tc-offload: off
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> shaper /etc # ethtool -k eth3.777
> Features for eth3.777:
> rx-checksumming: off [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: on
> tx-checksum-sctp: on
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [requested on]
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: off
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: off
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on
> rx-vlan-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: on [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: on
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-sctp-segmentation: on
> tx-esp-segmentation: off [fixed]
> fcoe-mtu: off [requested on]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off [fixed]
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
>
>
> shaper /etc # ethtool -i eth3
> driver: ixgbe
> version: 5.0.0-k
> firmware-version: 0x2b2c0001
> expansion-rom-version:
> bus-info: 0000:06:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> mpstat -P ALL 10
> Linux 4.12.9-build-0132 (shaper) 09/13/17 _x86_64_ (32 CPU)
>
> 18:15:55 CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %idle
> 18:16:05 all 0.00 0.00 0.00 0.00 0.00 0.63
> 0.00 0.00 99.37
> 18:16:05 0 0.00 0.00 0.13 0.00 0.00 0.13
> 0.00 0.00 99.73
> 18:16:05 1 0.00 0.00 0.00 0.00 0.00 1.34
> 0.00 0.00 98.66
> 18:16:05 2 0.00 0.00 0.00 0.00 0.00 1.32
> 0.00 0.00 98.68
> 18:16:05 3 0.00 0.00 0.00 0.00 0.00 1.81
> 0.00 0.00 98.19
> 18:16:05 4 0.00 0.00 0.00 0.00 0.00 1.78
> 0.00 0.00 98.22
> 18:16:05 5 0.00 0.00 0.00 0.00 0.00 1.77
> 0.00 0.00 98.23
> 18:16:05 6 0.00 0.00 0.00 0.00 0.00 1.77
> 0.00 0.00 98.23
> 18:16:05 7 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 8 0.00 0.00 0.00 0.00 0.00 0.15
> 0.00 0.00 99.85
> 18:16:05 9 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 10 0.00 0.00 0.00 0.00 0.00 0.15
> 0.00 0.00 99.85
> 18:16:05 11 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 12 0.00 0.00 0.00 0.00 0.00 0.48
> 0.00 0.00 99.52
> 18:16:05 13 0.00 0.00 0.00 0.00 0.00 0.59
> 0.00 0.00 99.41
> 18:16:05 14 0.00 0.00 0.00 0.00 0.00 1.30
> 0.00 0.00 98.70
> 18:16:05 15 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 16 0.00 0.00 0.00 0.00 0.00 0.12
> 0.00 0.00 99.88
> 18:16:05 17 0.00 0.00 0.00 0.00 0.00 0.82
> 0.00 0.00 99.18
> 18:16:05 18 0.00 0.00 0.00 0.00 0.00 0.71
> 0.00 0.00 99.29
> 18:16:05 19 0.00 0.00 0.00 0.00 0.00 0.71
> 0.00 0.00 99.29
> 18:16:05 20 0.00 0.00 0.00 0.00 0.00 0.72
> 0.00 0.00 99.28
> 18:16:05 21 0.00 0.00 0.00 0.00 0.00 0.59
> 0.00 0.00 99.41
> 18:16:05 22 0.00 0.00 0.00 0.00 0.00 1.30
> 0.00 0.00 98.70
> 18:16:05 23 0.00 0.00 0.00 0.00 0.00 1.18
> 0.00 0.00 98.82
> 18:16:05 24 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 25 0.00 0.00 0.00 0.00 0.00 0.84
> 0.00 0.00 99.16
> 18:16:05 26 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 27 0.00 0.00 0.00 0.00 0.00 0.85
> 0.00 0.00 99.15
> 18:16:05 28 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 29 0.00 0.00 0.12 0.00 0.00 0.12
> 0.00 0.00 99.76
> 18:16:05 30 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> 18:16:05 31 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 15:20 HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Denys Fedoryshchenko
2017-09-13 15:34 ` Denys Fedoryshchenko
@ 2017-09-13 15:51 ` Eric Dumazet
2017-09-13 15:56 ` Denys Fedoryshchenko
1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 15:51 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers
On Wed, 2017-09-13 at 18:20 +0300, Denys Fedoryshchenko wrote:
> Hi,
>
> I noticed after increasing bandwidth over some amount HTB started to
> throttle classes it should not throttle.
> Also estimated rate in htb totally wrong, while byte counters is
> correct.
>
> Is there any overflow or something?
Thanks Denys for the report, I will take a look at this, since I
probably introduced some regression.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 15:51 ` HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Eric Dumazet
@ 2017-09-13 15:56 ` Denys Fedoryshchenko
0 siblings, 0 replies; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 15:56 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Network Developers, netdev-owner
On 2017-09-13 18:51, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 18:20 +0300, Denys Fedoryshchenko wrote:
>> Hi,
>>
>> I noticed after increasing bandwidth over some amount HTB started to
>> throttle classes it should not throttle.
>> Also estimated rate in htb totally wrong, while byte counters is
>> correct.
>>
>> Is there any overflow or something?
>
> Thanks Denys for the report, I will take a look at this, since I
> probably introduced some regression.
It's definitely not something recent, this system was on older kernel
with uptime over 200 days, and this bottleneck was present, i noticed it
long time before.
But never tried to remove estimators (increasing burst/cburst to insane
values saved me for a while).
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 15:34 ` Denys Fedoryshchenko
@ 2017-09-13 16:16 ` Eric Dumazet
2017-09-13 16:27 ` Denys Fedoryshchenko
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 16:16 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers
On Wed, 2017-09-13 at 18:34 +0300, Denys Fedoryshchenko wrote:
> Well, probably i am answering my own question, removing estimator from
> classes seems drastically improve situation.
> It seems estimator has some issues that cause shaper to behave
> incorrectly (throttling traffic while it should not).
> But i guess thats a bug?
> As i was not able to predict such bottleneck by CPU load measurements.
Well, there was a reason we disabled HTB class estimators by default ;)
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=64153ce0a7b61b2a5cacb01805cbf670142339e9
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 16:16 ` Eric Dumazet
@ 2017-09-13 16:27 ` Denys Fedoryshchenko
2017-09-13 16:42 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 16:27 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Network Developers
On 2017-09-13 19:16, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 18:34 +0300, Denys Fedoryshchenko wrote:
>> Well, probably i am answering my own question, removing estimator from
>> classes seems drastically improve situation.
>> It seems estimator has some issues that cause shaper to behave
>> incorrectly (throttling traffic while it should not).
>> But i guess thats a bug?
>> As i was not able to predict such bottleneck by CPU load measurements.
>
> Well, there was a reason we disabled HTB class estimators by default ;)
>
>
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=64153ce0a7b61b2a5cacb01805cbf670142339e9
As soon as disabling it solve my problem - i'm fine, hehe, but i guess
other people who might hit this problem, should be aware how to find
reason.
They should not be disappointed in Linux :)
Because i can't measure this bottleneck before it happens, i'm seeing on
mpstat all cpu's are idle, and same time traffic is throttled.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 16:27 ` Denys Fedoryshchenko
@ 2017-09-13 16:42 ` Eric Dumazet
2017-09-13 16:55 ` Eric Dumazet
2017-09-13 18:16 ` [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples Eric Dumazet
0 siblings, 2 replies; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 16:42 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers
On Wed, 2017-09-13 at 19:27 +0300, Denys Fedoryshchenko wrote:
> On 2017-09-13 19:16, Eric Dumazet wrote:
> > On Wed, 2017-09-13 at 18:34 +0300, Denys Fedoryshchenko wrote:
> >> Well, probably i am answering my own question, removing estimator from
> >> classes seems drastically improve situation.
> >> It seems estimator has some issues that cause shaper to behave
> >> incorrectly (throttling traffic while it should not).
> >> But i guess thats a bug?
> >> As i was not able to predict such bottleneck by CPU load measurements.
> >
> > Well, there was a reason we disabled HTB class estimators by default ;)
> >
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=64153ce0a7b61b2a5cacb01805cbf670142339e9
>
> As soon as disabling it solve my problem - i'm fine, hehe, but i guess
> other people who might hit this problem, should be aware how to find
> reason.
> They should not be disappointed in Linux :)
Well, if they enable rate estimators while kernel does not set them by
default, they get what they want, at a cost.
> Because i can't measure this bottleneck before it happens, i'm seeing on
> mpstat all cpu's are idle, and same time traffic is throttled.
Normally things were supposed to get much better in linux-4.10
( https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=1c0d32fde5bdf1184bc274f864c09799278a1114 )
But I apparently added a scaling bug.
I will try :
diff --git a/net/core/gen_estimator.c b/net/core/gen_estimator.c
index 0385dece1f6fe5e26df1ce5f40956a79a2eebbf4..7c1ffd6f950172c1915d8e5fa2b5e3f77e4f4c78 100644
--- a/net/core/gen_estimator.c
+++ b/net/core/gen_estimator.c
@@ -83,10 +83,10 @@ static void est_timer(unsigned long arg)
u64 rate, brate;
est_fetch_counters(est, &b);
- brate = (b.bytes - est->last_bytes) << (8 - est->ewma_log);
+ brate = (b.bytes - est->last_bytes) << (10 - est->ewma_log - est->intvl_log);
brate -= (est->avbps >> est->ewma_log);
- rate = (u64)(b.packets - est->last_packets) << (8 - est->ewma_log);
+ rate = (u64)(b.packets - est->last_packets) << (10 - est->ewma_log - est->intvl_log);
rate -= (est->avpps >> est->ewma_log);
write_seqcount_begin(&est->seq);
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 16:42 ` Eric Dumazet
@ 2017-09-13 16:55 ` Eric Dumazet
2017-09-13 17:12 ` Denys Fedoryshchenko
2017-09-13 18:16 ` [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples Eric Dumazet
1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 16:55 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers
On Wed, 2017-09-13 at 09:42 -0700, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 19:27 +0300, Denys Fedoryshchenko wrote:
> > On 2017-09-13 19:16, Eric Dumazet wrote:
> > > On Wed, 2017-09-13 at 18:34 +0300, Denys Fedoryshchenko wrote:
> > >> Well, probably i am answering my own question, removing estimator from
> > >> classes seems drastically improve situation.
> > >> It seems estimator has some issues that cause shaper to behave
> > >> incorrectly (throttling traffic while it should not).
> > >> But i guess thats a bug?
> > >> As i was not able to predict such bottleneck by CPU load measurements.
> > >
> > > Well, there was a reason we disabled HTB class estimators by default ;)
> > >
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=64153ce0a7b61b2a5cacb01805cbf670142339e9
> >
> > As soon as disabling it solve my problem - i'm fine, hehe, but i guess
> > other people who might hit this problem, should be aware how to find
> > reason.
> > They should not be disappointed in Linux :)
>
> Well, if they enable rate estimators while kernel does not set them by
> default, they get what they want, at a cost.
>
> > Because i can't measure this bottleneck before it happens, i'm seeing on
> > mpstat all cpu's are idle, and same time traffic is throttled.
>
> Normally things were supposed to get much better in linux-4.10
>
> ( https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=1c0d32fde5bdf1184bc274f864c09799278a1114 )
>
> But I apparently added a scaling bug.
>
> I will try :
>
> diff --git a/net/core/gen_estimator.c b/net/core/gen_estimator.c
> index 0385dece1f6fe5e26df1ce5f40956a79a2eebbf4..7c1ffd6f950172c1915d8e5fa2b5e3f77e4f4c78 100644
> --- a/net/core/gen_estimator.c
> +++ b/net/core/gen_estimator.c
> @@ -83,10 +83,10 @@ static void est_timer(unsigned long arg)
> u64 rate, brate;
>
> est_fetch_counters(est, &b);
> - brate = (b.bytes - est->last_bytes) << (8 - est->ewma_log);
> + brate = (b.bytes - est->last_bytes) << (10 - est->ewma_log - est->intvl_log);
> brate -= (est->avbps >> est->ewma_log);
>
> - rate = (u64)(b.packets - est->last_packets) << (8 - est->ewma_log);
> + rate = (u64)(b.packets - est->last_packets) << (10 - est->ewma_log - est->intvl_log);
> rate -= (est->avpps >> est->ewma_log);
>
> write_seqcount_begin(&est->seq);
Much better indeed
# tc -s -d class sh dev eth0 classid 7002:11 ; sleep 10 ;tc -s -d class
sh dev eth0 classid 7002:11
class htb 7002:11 parent 7002:1 prio 5 quantum 200000 rate 5Gbit ceil
5Gbit linklayer ethernet burst 80000b/1 mpu 0b cburst 80000b/1 mpu 0b
level 0 rate_handle 1
Sent 389085117074 bytes 256991500 pkt (dropped 0, overlimits 5926926
requeues 0)
rate 4999Mbit 412762pps backlog 136260b 2p requeues 0
TCP pkts/rtx 256991584/0 bytes 389085252840/0
lended: 5961250 borrowed: 0 giants: 0
tokens: -1664 ctokens: -1664
class htb 7002:11 parent 7002:1 prio 5 quantum 200000 rate 5Gbit ceil
5Gbit linklayer ethernet burst 80000b/1 mpu 0b cburst 80000b/1 mpu 0b
level 0 rate_handle 1
Sent 395336315580 bytes 261120429 pkt (dropped 0, overlimits 6021776
requeues 0)
rate 4999Mbit 412788pps backlog 68Kb 2p requeues 0
TCP pkts/rtx 261120469/0 bytes 395336384730/0
lended: 6056793 borrowed: 0 giants: 0
tokens: -1478 ctokens: -1478
echo "(395336315580-389085117074)/10*8" | bc
5000958800
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 16:55 ` Eric Dumazet
@ 2017-09-13 17:12 ` Denys Fedoryshchenko
2017-09-13 17:20 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 17:12 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Network Developers, netdev-owner
On 2017-09-13 19:55, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 09:42 -0700, Eric Dumazet wrote:
>> On Wed, 2017-09-13 at 19:27 +0300, Denys Fedoryshchenko wrote:
>> > On 2017-09-13 19:16, Eric Dumazet wrote:
>> > > On Wed, 2017-09-13 at 18:34 +0300, Denys Fedoryshchenko wrote:
>> > >> Well, probably i am answering my own question, removing estimator from
>> > >> classes seems drastically improve situation.
>> > >> It seems estimator has some issues that cause shaper to behave
>> > >> incorrectly (throttling traffic while it should not).
>> > >> But i guess thats a bug?
>> > >> As i was not able to predict such bottleneck by CPU load measurements.
>> > >
>> > > Well, there was a reason we disabled HTB class estimators by default ;)
>> > >
>> > >
>> > > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=64153ce0a7b61b2a5cacb01805cbf670142339e9
>> >
>> > As soon as disabling it solve my problem - i'm fine, hehe, but i guess
>> > other people who might hit this problem, should be aware how to find
>> > reason.
>> > They should not be disappointed in Linux :)
>>
>> Well, if they enable rate estimators while kernel does not set them by
>> default, they get what they want, at a cost.
>>
>> > Because i can't measure this bottleneck before it happens, i'm seeing on
>> > mpstat all cpu's are idle, and same time traffic is throttled.
>>
>> Normally things were supposed to get much better in linux-4.10
>>
>> (
>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=1c0d32fde5bdf1184bc274f864c09799278a1114
>> )
>>
>> But I apparently added a scaling bug.
>>
>> I will try :
>>
>> diff --git a/net/core/gen_estimator.c b/net/core/gen_estimator.c
>> index
>> 0385dece1f6fe5e26df1ce5f40956a79a2eebbf4..7c1ffd6f950172c1915d8e5fa2b5e3f77e4f4c78
>> 100644
>> --- a/net/core/gen_estimator.c
>> +++ b/net/core/gen_estimator.c
>> @@ -83,10 +83,10 @@ static void est_timer(unsigned long arg)
>> u64 rate, brate;
>>
>> est_fetch_counters(est, &b);
>> - brate = (b.bytes - est->last_bytes) << (8 - est->ewma_log);
>> + brate = (b.bytes - est->last_bytes) << (10 - est->ewma_log -
>> est->intvl_log);
>> brate -= (est->avbps >> est->ewma_log);
>>
>> - rate = (u64)(b.packets - est->last_packets) << (8 -
>> est->ewma_log);
>> + rate = (u64)(b.packets - est->last_packets) << (10 -
>> est->ewma_log - est->intvl_log);
>> rate -= (est->avpps >> est->ewma_log);
>>
>> write_seqcount_begin(&est->seq);
>
>
> Much better indeed
>
> # tc -s -d class sh dev eth0 classid 7002:11 ; sleep 10 ;tc -s -d class
> sh dev eth0 classid 7002:11
>
> class htb 7002:11 parent 7002:1 prio 5 quantum 200000 rate 5Gbit ceil
> 5Gbit linklayer ethernet burst 80000b/1 mpu 0b cburst 80000b/1 mpu 0b
> level 0 rate_handle 1
> Sent 389085117074 bytes 256991500 pkt (dropped 0, overlimits 5926926
> requeues 0)
> rate 4999Mbit 412762pps backlog 136260b 2p requeues 0
> TCP pkts/rtx 256991584/0 bytes 389085252840/0
> lended: 5961250 borrowed: 0 giants: 0
> tokens: -1664 ctokens: -1664
>
> class htb 7002:11 parent 7002:1 prio 5 quantum 200000 rate 5Gbit ceil
> 5Gbit linklayer ethernet burst 80000b/1 mpu 0b cburst 80000b/1 mpu 0b
> level 0 rate_handle 1
> Sent 395336315580 bytes 261120429 pkt (dropped 0, overlimits 6021776
> requeues 0)
> rate 4999Mbit 412788pps backlog 68Kb 2p requeues 0
> TCP pkts/rtx 261120469/0 bytes 395336384730/0
> lended: 6056793 borrowed: 0 giants: 0
> tokens: -1478 ctokens: -1478
>
>
> echo "(395336315580-389085117074)/10*8" | bc
> 5000958800
For my case, as load increased now, i am hitting same issue (i tried to
play with quantum / bursts as well, didnt helped):
tc -s -d class show dev eth3.777 classid 1:111;sleep 5;tc -s -d class
show dev eth3.777 classid 1:111
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst 100000b/1
mpu 0b level 0
Sent 864151559 bytes 730566 pkt (dropped 15111, overlimits 0 requeues
0)
backlog 73968000b 39934p requeues 0
lended: 499867 borrowed: 0 giants: 0
tokens: 608 ctokens: 121
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst 100000b/1
mpu 0b level 0
Sent 1469352160 bytes 1243649 pkt (dropped 42933, overlimits 0 requeues
0)
backlog 82536047b 39963p requeues 0
lended: 810475 borrowed: 0 giants: 0
tokens: 612 ctokens: 122
(1469352160-864151559)/5*8
968320961.60000000000000000000
Less than 1Gbit and it's being throttled
Total bandwidth:
class htb 1:1 root rate 100Gbit ceil 100Gbit linklayer ethernet burst
100000b/1 mpu 0b cburst 100000b/1 mpu 0b level 7
Sent 7839730635 bytes 8537393 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: 123 ctokens: 123
class htb 1:1 root rate 100Gbit ceil 100Gbit linklayer ethernet burst
100000b/1 mpu 0b cburst 100000b/1 mpu 0b level 7
Sent 11043190453 bytes 12008366 pkt (dropped 0, overlimits 0 requeues
0)
backlog 0b 0p requeues 0
lended: 0 borrowed: 0 giants: 0
tokens: 124 ctokens: 124
694kpps
5.1Gbit
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 17:12 ` Denys Fedoryshchenko
@ 2017-09-13 17:20 ` Eric Dumazet
2017-09-13 17:35 ` Denys Fedoryshchenko
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 17:20 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers, netdev-owner
On Wed, 2017-09-13 at 20:12 +0300, Denys Fedoryshchenko wrote:
> For my case, as load increased now, i am hitting same issue (i tried to
> play with quantum / bursts as well, didnt helped):
>
> tc -s -d class show dev eth3.777 classid 1:111;sleep 5;tc -s -d class
> show dev eth3.777 classid 1:111
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
> ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst 100000b/1
> mpu 0b level 0
> Sent 864151559 bytes 730566 pkt (dropped 15111, overlimits 0 requeues
> 0)
> backlog 73968000b 39934p requeues 0
> lended: 499867 borrowed: 0 giants: 0
> tokens: 608 ctokens: 121
>
You have drops (and ~40,000 packets in backlog)
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
> ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst 100000b/1
> mpu 0b level 0
> Sent 1469352160 bytes 1243649 pkt (dropped 42933, overlimits 0 requeues
> 0)
> backlog 82536047b 39963p requeues 0
> lended: 810475 borrowed: 0 giants: 0
> tokens: 612 ctokens: 122
>
> (1469352160-864151559)/5*8
> 968320961.60000000000000000000
> Less than 1Gbit and it's being throttled
It is not : "overlimits 0" means this class was not throttled.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 17:20 ` Eric Dumazet
@ 2017-09-13 17:35 ` Denys Fedoryshchenko
2017-09-13 17:58 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Denys Fedoryshchenko @ 2017-09-13 17:35 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Kernel Network Developers, netdev-owner
On 2017-09-13 20:20, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 20:12 +0300, Denys Fedoryshchenko wrote:
>
>> For my case, as load increased now, i am hitting same issue (i tried
>> to
>> play with quantum / bursts as well, didnt helped):
>>
>> tc -s -d class show dev eth3.777 classid 1:111;sleep 5;tc -s -d class
>> show dev eth3.777 classid 1:111
>> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
>> ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst
>> 100000b/1
>> mpu 0b level 0
>> Sent 864151559 bytes 730566 pkt (dropped 15111, overlimits 0
>> requeues
>> 0)
>> backlog 73968000b 39934p requeues 0
>> lended: 499867 borrowed: 0 giants: 0
>> tokens: 608 ctokens: 121
>>
>
> You have drops (and ~40,000 packets in backlog)
>
>
>> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 20Gbit
>> ceil 100Gbit linklayer ethernet burst 100000b/1 mpu 0b cburst
>> 100000b/1
>> mpu 0b level 0
>> Sent 1469352160 bytes 1243649 pkt (dropped 42933, overlimits 0
>> requeues
>> 0)
>> backlog 82536047b 39963p requeues 0
>> lended: 810475 borrowed: 0 giants: 0
>> tokens: 612 ctokens: 122
>>
>> (1469352160-864151559)/5*8
>> 968320961.60000000000000000000
>> Less than 1Gbit and it's being throttled
>
> It is not : "overlimits 0" means this class was not throttled.
Overlimits never appear in HTB as i know, here is simulation on this
class that have constant "at least" 1G traffic, i throttled it to 1Kbit
to simulate forced drops:
shapernew ~ # sh /etc/shaper.cfg;sleep 1;tc -s -d class show dev
eth3.777 classid 1:111;tc qdisc del dev eth3.777 root
class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 1Kbit
ceil 1Kbit linklayer ethernet burst 31280b/1 mpu 0b cburst 31280b/1 mpu
0b level 0
Sent 134350019 bytes 117520 pkt (dropped 7819, overlimits 0 requeues 0)
backlog 7902126b 4976p requeues 0
lended: 86694 borrowed: 0 giants: 0
tokens: -937500000 ctokens: -937500000
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 17:35 ` Denys Fedoryshchenko
@ 2017-09-13 17:58 ` Eric Dumazet
2017-09-13 18:03 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 17:58 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers, netdev-owner
On Wed, 2017-09-13 at 20:35 +0300, Denys Fedoryshchenko wrote:
> Overlimits never appear in HTB as i know, here is simulation on this
> class that have constant "at least" 1G traffic, i throttled it to 1Kbit
> to simulate forced drops:
>
> shapernew ~ # sh /etc/shaper.cfg;sleep 1;tc -s -d class show dev
> eth3.777 classid 1:111;tc qdisc del dev eth3.777 root
> class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 1Kbit
> ceil 1Kbit linklayer ethernet burst 31280b/1 mpu 0b cburst 31280b/1 mpu
> 0b level 0
> Sent 134350019 bytes 117520 pkt (dropped 7819, overlimits 0 requeues 0)
> backlog 7902126b 4976p requeues 0
> lended: 86694 borrowed: 0 giants: 0
> tokens: -937500000 ctokens: -937500000
>
Oh right, I am using a local patch for this. Time to upstream :/
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well)
2017-09-13 17:58 ` Eric Dumazet
@ 2017-09-13 18:03 ` Eric Dumazet
0 siblings, 0 replies; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 18:03 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: Linux Kernel Network Developers, netdev-owner
On Wed, 2017-09-13 at 10:58 -0700, Eric Dumazet wrote:
> On Wed, 2017-09-13 at 20:35 +0300, Denys Fedoryshchenko wrote:
>
> > Overlimits never appear in HTB as i know, here is simulation on this
> > class that have constant "at least" 1G traffic, i throttled it to 1Kbit
> > to simulate forced drops:
> >
> > shapernew ~ # sh /etc/shaper.cfg;sleep 1;tc -s -d class show dev
> > eth3.777 classid 1:111;tc qdisc del dev eth3.777 root
> > class htb 1:111 parent 1:1 leaf 111: prio 0 quantum 50000 rate 1Kbit
> > ceil 1Kbit linklayer ethernet burst 31280b/1 mpu 0b cburst 31280b/1 mpu
> > 0b level 0
> > Sent 134350019 bytes 117520 pkt (dropped 7819, overlimits 0 requeues 0)
> > backlog 7902126b 4976p requeues 0
> > lended: 86694 borrowed: 0 giants: 0
> > tokens: -937500000 ctokens: -937500000
> >
>
> Oh right, I am using a local patch for this. Time to upstream :/
Please try :
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 7e148376ba528efabe5a53a09653f9161c264be7..c6d7ae81b41f4e277afb93a3003fefcd3f27de35 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -142,6 +142,7 @@ struct htb_class {
struct rb_node node[TC_HTB_NUMPRIO]; /* node for self or feed tree */
unsigned int drops ____cacheline_aligned_in_smp;
+ unsigned int overlimits;
};
struct htb_level {
@@ -533,6 +534,9 @@ htb_change_class_mode(struct htb_sched *q, struct htb_class *cl, s64 *diff)
if (new_mode == cl->cmode)
return;
+ if (new_mode == HTB_CANT_SEND)
+ cl->overlimits++;
+
if (cl->prio_activity) { /* not necessary: speed optimization */
if (cl->cmode != HTB_CANT_SEND)
htb_deactivate_prios(q, cl);
@@ -1143,6 +1147,7 @@ htb_dump_class_stats(struct Qdisc *sch, unsigned long arg, struct gnet_dump *d)
struct htb_class *cl = (struct htb_class *)arg;
struct gnet_stats_queue qs = {
.drops = cl->drops,
+ .overlimits = cl->overlimits,
};
__u32 qlen = 0;
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples
2017-09-13 16:42 ` Eric Dumazet
2017-09-13 16:55 ` Eric Dumazet
@ 2017-09-13 18:16 ` Eric Dumazet
2017-09-13 20:34 ` David Miller
1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-09-13 18:16 UTC (permalink / raw)
To: Denys Fedoryshchenko
Cc: David Miller, netdev, Jamal Hadi Salim, Cong Wang, Jiri Pirko
From: Eric Dumazet <edumazet@google.com>
Denys reported wrong rate estimations with HTB classes.
It appears the bug was added in linux-4.10, since my tests
where using intervals of one second only.
HTB using 4 sec default rate estimators, reported rates
were 4x higher.
We need to properly scale the bytes/packets samples before
integrating them in EWMA.
Tested:
echo 1 >/sys/module/sch_htb/parameters/htb_rate_est
Setup HTB with one class with a rate/cail of 5Gbit
Generate traffic on this class
tc -s -d cl sh dev eth0 classid 7002:11
class htb 7002:11 parent 7002:1 prio 5 quantum 200000 rate 5Gbit ceil
5Gbit linklayer ethernet burst 80000b/1 mpu 0b cburst 80000b/1 mpu 0b
level 0 rate_handle 1
Sent 1488215421648 bytes 982969243 pkt (dropped 0, overlimits 0
requeues 0)
rate 5Gbit 412814pps backlog 136260b 2p requeues 0
TCP pkts/rtx 982969327/45 bytes 1488215557414/68130
lended: 22732826 borrowed: 0 giants: 0
tokens: -1684 ctokens: -1684
Fixes: 1c0d32fde5bd ("net_sched: gen_estimator: complete rewrite of rate estimators")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Denys Fedoryshchenko <nuclearcat@nuclearcat.com>
---
net/core/gen_estimator.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/net/core/gen_estimator.c b/net/core/gen_estimator.c
index 0385dece1f6fe5e26df1ce5f40956a79a2eebbf4..7c1ffd6f950172c1915d8e5fa2b5e3f77e4f4c78 100644
--- a/net/core/gen_estimator.c
+++ b/net/core/gen_estimator.c
@@ -83,10 +83,10 @@ static void est_timer(unsigned long arg)
u64 rate, brate;
est_fetch_counters(est, &b);
- brate = (b.bytes - est->last_bytes) << (8 - est->ewma_log);
+ brate = (b.bytes - est->last_bytes) << (10 - est->ewma_log - est->intvl_log);
brate -= (est->avbps >> est->ewma_log);
- rate = (u64)(b.packets - est->last_packets) << (8 - est->ewma_log);
+ rate = (u64)(b.packets - est->last_packets) << (10 - est->ewma_log - est->intvl_log);
rate -= (est->avpps >> est->ewma_log);
write_seqcount_begin(&est->seq);
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples
2017-09-13 18:16 ` [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples Eric Dumazet
@ 2017-09-13 20:34 ` David Miller
0 siblings, 0 replies; 15+ messages in thread
From: David Miller @ 2017-09-13 20:34 UTC (permalink / raw)
To: eric.dumazet; +Cc: nuclearcat, netdev, jhs, xiyou.wangcong, jiri
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed, 13 Sep 2017 11:16:45 -0700
> From: Eric Dumazet <edumazet@google.com>
>
> Denys reported wrong rate estimations with HTB classes.
>
> It appears the bug was added in linux-4.10, since my tests
> where using intervals of one second only.
>
> HTB using 4 sec default rate estimators, reported rates
> were 4x higher.
>
> We need to properly scale the bytes/packets samples before
> integrating them in EWMA.
>
> Tested:
...
> Fixes: 1c0d32fde5bd ("net_sched: gen_estimator: complete rewrite of rate estimators")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Reported-by: Denys Fedoryshchenko <nuclearcat@nuclearcat.com>
Applied and queued up for -stable, thanks Eric.
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2017-09-13 20:34 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-13 15:20 HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Denys Fedoryshchenko
2017-09-13 15:34 ` Denys Fedoryshchenko
2017-09-13 16:16 ` Eric Dumazet
2017-09-13 16:27 ` Denys Fedoryshchenko
2017-09-13 16:42 ` Eric Dumazet
2017-09-13 16:55 ` Eric Dumazet
2017-09-13 17:12 ` Denys Fedoryshchenko
2017-09-13 17:20 ` Eric Dumazet
2017-09-13 17:35 ` Denys Fedoryshchenko
2017-09-13 17:58 ` Eric Dumazet
2017-09-13 18:03 ` Eric Dumazet
2017-09-13 18:16 ` [PATCH net] net_sched: gen_estimator: fix scaling error in bytes/packets samples Eric Dumazet
2017-09-13 20:34 ` David Miller
2017-09-13 15:51 ` HTB going crazy over ~5Gbit/s (4.12.9, but problem present in older kernels as well) Eric Dumazet
2017-09-13 15:56 ` Denys Fedoryshchenko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).