Turning on TC throttles my HTTP downloads for no known reason

Turning on TC throttles my HTTP downloads for no known reason

[Alan McKay]

Hi folks,

I'm using Shorewall to configure my firewall, and recently turned on
traffic shaping ( TC_ENABLED=Internal in shorewall.conf) because I
introduced a VOIP phone in my home network.  The VOIP calls had been
getting clobbered when big downloads were going on, so I had to enable
QoS.   This fixed the VOIP issue and all was well and good.

However then I started to notice that my connection was a lot slower
in general.  So I wrote a program to use the speedtest.net facilities
to log my connection speed and graph it.  Sure enough, when I turn on
TC my HTTP downloads from my ISP's speedtest server (within their
network) are in the 5Mbit/s range.  As soon as I turn off TC this
jumps up to the advertise 25Mbit or so range with peaks even at
55Mbit.   You can see my graphs here :

http://108.162.170.186/speedtest/

And here is one chart I saved out so it won't get wiped out by my updates :
http://108.162.170.186/speedtest/downloadday20130330.png

You can see the big dip on the right is when I had TC turned on.

So the interesting part is this ... I reported this to the Shorewall
users list, and Tom immediately said "Oh, have you looked at FAQ
97a?".   So I went and had a look :

---snip---

(FAQ 97a) I enable Shorewall traffic shaping and now my download rate
is way below what I specified

Answer: This is likely due to Generic Receive Offload (GRO) being
enabled in the network adapter. To verify, install the ethtool package
and use the -k command:

root@gateway:/etc/shorewall# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
root@gateway:/etc/shorewall#

To work around the issue, use this command:

ethtool -K ethN gro off

---snip---

So I turned that off and immediately it fixed my problem!  Fantastic!
 I then added that setting to post-up in /etc/network/interfaces on my
Ubuntu 12.10 firewall, and restarted networking to test that it still
worked.   And it did.   Great!

Until I had to reboot my firewall (right where you see the big dip on
that fixed graph).    After the reboot everything came up great and TC
was enabled as expected, and this gro setting was still off as
expected.   But my downloads were getting clobbered again and were in
the 5 Mbit range.

And as soon as I turned off TC they shoot back up again.

I've reported all this to the shorewall list and everyone there seems stumped.

Here is how Shorewall configures the TC on my eth0 (internet facing
NIC on my firewall).    Some of the commands here are shorewallisms
but I think this should still be pretty straightforward.

        qt $TC qdisc del dev eth0 root
        qt $TC qdisc del dev eth0 ingress
        eth0_mtu=$(get_device_mtu eth0)
        eth0_mtu1=$(get_device_mtu1 eth0)
        run_tc qdisc add dev eth0 root handle 1: htb default 13 r2q 5
        run_tc class add dev eth0 parent 1: classid 1:1 htb rate
1000kbit $eth0_mtu1
        run_tc qdisc add dev eth0 handle ffff: ingress
        run_tc filter add dev eth0 parent ffff: protocol all prio 10 basic \
            police mpu 64 rate 35000kbit burst 10kb action drop

        [ $eth0_mtu -gt 2500 ] && quantum=$eth0_mtu || quantum%00
        run_tc class add dev eth0 parent 1:1 classid 1:11 htb rate
100kbit ceil 180kbit prio 1 $eth0_mtu1 quantum $quantum
        run_tc qdisc add dev eth0 parent 1:11 handle 3: sfq quantum
$quantum limit 127 perturb 10
        run_tc filter add dev eth0 protocol all parent 1:0 prio 276
handle 1 fw classid 1:11
        run_tc filter add dev eth0 parent 1:0 protocol ip prio 266 u32
match ip tos 0x68 0xfc flowid 1:11
        run_tc filter add dev eth0 parent 1:0 protocol ip prio 266 u32
match ip tos 0xb8 0xfc flowid 1:11
        progress_message "   TC Class 1:11 defined."

        [ $eth0_mtu -gt 12500 ] && quantum=$eth0_mtu || quantum\x12500
        run_tc class add dev eth0 parent 1:1 classid 1:12 htb rate
500kbit ceil 1000kbit prio 2 $eth0_mtu1 quantum $quantum
        run_tc qdisc add dev eth0 parent 1:12 handle 4: sfq quantum
$quantum limit 127 perturb 10
        run_tc filter add dev eth0 protocol all parent 1:0 prio 532
handle 2 fw classid 1:12
        run_tc filter add dev eth0 parent 1:0 protocol ip prio 522 u32\
            match ip protocol 6 0xff\
            match u8 0x05 0x0f at 0\
            match u16 0x0000 0xffc0 at 2\
            match u8 0x10 0xff at 33 flowid 1:12
        run_tc filter add dev eth0 parent 1:0 protocol ip prio 522 u32
match ip tos 0x10 0x10 flowid 1:12
        progress_message "   TC Class 1:12 defined."

        [ $eth0_mtu -gt 12500 ] && quantum=$eth0_mtu || quantum\x12500
        run_tc class add dev eth0 parent 1:1 classid 1:13 htb rate
500kbit ceil 1000kbit prio 3 $eth0_mtu1 quantum $quantum
        run_tc qdisc add dev eth0 parent 1:13 handle 5: sfq quantum
$quantum limit 127 perturb 10
        run_tc filter add dev eth0 protocol all parent 1:0 prio 788
handle 3 fw classid 1:13
        progress_message "   TC Class 1:13 defined."

        [ $eth0_mtu -gt 6250 ] && quantum=$eth0_mtu || quantumb50
        run_tc class add dev eth0 parent 1:1 classid 1:14 htb rate
250kbit ceil 500kbit prio 4 $eth0_mtu1 quantum $quantum
        run_tc qdisc add dev eth0 parent 1:14 handle 6: sfq quantum
$quantum limit 127 perturb 10
        run_tc filter add dev eth0 protocol all parent 1:0 prio 1044
handle 4 fw classid 1:14
        progress_message "   TC Class 1:14 defined."









--
“Don't eat anything you've ever seen advertised on TV”
         - Michael Pollan, author of "In Defense of Food"

Re: Turning on TC throttles my HTTP downloads for no known reason

[Alan McKay]

On Sun, Mar 31, 2013 at 11:32 AM, Alan McKay <alan.mckay@gmail.com> wrote:
> And here is one chart I saved out so it won't get wiped out by my updates :
> http://108.162.170.186/speedtest/downloadday20130330.png

p.s. the values in my legend do not match the values on the graph -
the graph is right and the legend is wrong.  I'm still working on this
bug - this is my first time using RRD tool


--
“Don't eat anything you've ever seen advertised on TV”
         - Michael Pollan, author of "In Defense of Food"

Re: Turning on TC throttles my HTTP downloads for no known reason

[Andy Furniss]

Alan McKay wrote:

> So I turned that off and immediately it fixed my problem!  Fantastic!
>   I then added that setting to post-up in /etc/network/interfaces on my
> Ubuntu 12.10 firewall, and restarted networking to test that it still
> worked.   And it did.   Great!
>
> Until I had to reboot my firewall (right where you see the big dip on
> that fixed graph).    After the reboot everything came up great and TC
> was enabled as expected, and this gro setting was still off as
> expected.   But my downloads were getting clobbered again and were in
> the 5 Mbit range.
>
> And as soon as I turned off TC they shoot back up again.

Strange - I would try turning gro on and off again when running to see 
if that helps and there is some timing issue with doing it at boot.

> I've reported all this to the shorewall list and everyone there seems stumped.
>
> Here is how Shorewall configures the TC on my eth0 (internet facing
> NIC on my firewall).    Some of the commands here are shorewallisms
> but I think this should still be pretty straightforward.
>
>          qt $TC qdisc del dev eth0 root
>          qt $TC qdisc del dev eth0 ingress
>          eth0_mtu=$(get_device_mtu eth0)
>          eth0_mtu1=$(get_device_mtu1 eth0)
>          run_tc qdisc add dev eth0 root handle 1: htb default 13 r2q 5
>          run_tc class add dev eth0 parent 1: classid 1:1 htb rate
> 1000kbit $eth0_mtu1
>          run_tc qdisc add dev eth0 handle ffff: ingress
>          run_tc filter add dev eth0 parent ffff: protocol all prio 10 basic \
>              police mpu 64 rate 35000kbit burst 10kb action drop

Another way to work around gro would be to use the mtu param for the 
policer try 3100 or may be bigger - I don't know how many packets get 
aggregated for you.

There is a potential problem with using protocol all in that you will be 
sending arp to potentially be dropped. If you are just shaping ipv4 then 
protocol ip would be better.

The same applies to using "htb default 13" though in this example having 
reasonable bandwidth and sfq on 13 will save the day.

Re: Turning on TC throttles my HTTP downloads for no known reason

[Alan McKay]

Oh and here is this :

root@userver:/etc/shorewall# tc qdisc show
qdisc htb 1: dev eth0 root refcnt 2 r2q 5 default 13 direct_packets_stat 1
qdisc sfq 3: dev eth0 parent 1:11 limit 127p quantum 2500b depth 127
divisor 1024 perturb 10sec
qdisc sfq 4: dev eth0 parent 1:12 limit 127p quantum 12500b depth 127
divisor 1024 perturb 10sec
qdisc sfq 5: dev eth0 parent 1:13 limit 127p quantum 12500b depth 127
divisor 1024 perturb 10sec
qdisc sfq 6: dev eth0 parent 1:14 limit 127p quantum 6250b depth 127
divisor 1024 perturb 10sec
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc htb 2: dev eth2 root refcnt 2 r2q 5000 default 13 direct_packets_stat 1
qdisc sfq 7: dev eth2 parent 2:11 limit 127p quantum 1500b depth 127
divisor 1024 perturb 10sec
qdisc sfq 8: dev eth2 parent 2:12 limit 127p quantum 12500b depth 127
divisor 1024 perturb 10sec
qdisc sfq 9: dev eth2 parent 2:13 limit 127p quantum 12500b depth 127
divisor 1024 perturb 10sec
qdisc sfq a: dev eth2 parent 2:14 limit 127p quantum 6250b depth 127
divisor 1024 perturb 10sec
qdisc ingress ffff: dev eth2 parent ffff:fff1 ----------------


--
“Don't eat anything you've ever seen advertised on TV”
         - Michael Pollan, author of "In Defense of Food"