Can't get tc to limit network traffic

Can't get tc to limit network traffic

[Teresa e Junior]

I have asked this on http://unix.stackexchange.com/questions/154144/,
and even offered a bounty, but nobody was able to answer me (if you want
the bounty, go there and grab it fast ;)

Based on this section of the Linux Advanced Routing & Traffic Control
HOWTO: http://lartc.org/howto/lartc.ratelimit.single.html, I can't get
tc to limit the network speed in my computer.

Basically, I have 13 machines in the network, and this particular
machine sometimes steals all traffic available. The router is a Motorola
SurfBoard modem with a few routing capabilities and firewall. The
machine I want to limit the traffic is 192.168.0.5, and also the script
is being run from 192.168.0.5.

Here is my adaption of the commands in the HOWTO for
/etc/NetworkManager/dispatcher.d/:

#!/bin/sh -eu

# clear any previous queuing disciplines (qdisc)
tc qdisc del dev wlan0 root 2>/dev/null ||:

# add a cbq qdisc; see `man tc-cbq' for details
if [ $2 = up ]; then
     # set to a 3mbit interface for more precise calculations
     tc qdisc add dev wlan0 root handle 1: cbq avpkt 1000  \
         bandwidth 3mbit

     # leave 30KB (240kbps) to other machines in the network
     tc class add dev wlan0 parent 1: classid 1:1 cbq      \
         rate 2832kbit allot 1500 prio 5 bounded isolated

     # redirect all traffic on 192.168.0.5 to the previous class
     tc filter add dev wlan0 parent 1: protocol ip prio 16 \
         u32 match ip dst 192.168.0.5 flowid 1:1

     # change the hashing algorithm every 10s to avoid collisions
     tc qdisc add dev wlan0 parent 1:1 sfq perturb 10
fi

The problem is that I have tried setting 2832kbit to very small values
for testing (like 16kbit), but I still can browse the web at high speed.
The problem is not in NetworkManager, because I'm testing the script
manually. Also, tc doesn't return any errors.

I have found that by changing dst 192.168.0.5 to src 192.168.0.5, the
upload speed is reliably limited, but I still haven't figured how to get
the download speed to work, which is the most important for me.

The following approach I have taken from wondershaper:

tc qdisc  add dev wlan0 handle ffff: ingress
tc filter add dev wlan0 parent ffff: protocol ip prio 50 u32 match ip \
     dst 192.168.0.5 police rate 200kbps burst 10k drop flowid :1

But the problem with this one is that it is very unreliable, it results in
unstable download speeds (if set to 200, it downloads at ~80-120).

Thank you!
Teresa and Junior

Re: Can't get tc to limit network traffic

[Mike Schmidt]

Basically, you cannot control inbound traffic with tc, except through
policing, which is what the wondershaper rule does. Policing does not
shape traffic, it simply drops the excess, which is why it is so
unreliable.  That is why the throttling works when only you set the
src to 192.168.0.5. The solution is to use an intermediate device,
like ifb, or imq, depending on the distribution you are running.  A
typical example is here:
http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb

I use ifb in our company's software, where we have a number of subnets
and or vlans to deal with, and ifb+hfsc has provided good results. I
am not familiar with cbq, although we did use htb, which is somewhat
similar, until it crashed the kernel under too much load. This problem
was fixed in later kernels (>2.6.36).  However, hfsc is much more
precise.

On Sun, Sep 14, 2014 at 9:37 PM, Teresa e Junior
<teresaejunior@gmail.com> wrote:
> I have asked this on http://unix.stackexchange.com/questions/154144/,
> and even offered a bounty, but nobody was able to answer me (if you want
> the bounty, go there and grab it fast ;)
>
> Based on this section of the Linux Advanced Routing & Traffic Control
> HOWTO: http://lartc.org/howto/lartc.ratelimit.single.html, I can't get
> tc to limit the network speed in my computer.
>
> Basically, I have 13 machines in the network, and this particular
> machine sometimes steals all traffic available. The router is a Motorola
> SurfBoard modem with a few routing capabilities and firewall. The
> machine I want to limit the traffic is 192.168.0.5, and also the script
> is being run from 192.168.0.5.
>
> Here is my adaption of the commands in the HOWTO for
> /etc/NetworkManager/dispatcher.d/:
>
> #!/bin/sh -eu
>
> # clear any previous queuing disciplines (qdisc)
> tc qdisc del dev wlan0 root 2>/dev/null ||:
>
> # add a cbq qdisc; see `man tc-cbq' for details
> if [ $2 = up ]; then
>     # set to a 3mbit interface for more precise calculations
>     tc qdisc add dev wlan0 root handle 1: cbq avpkt 1000  \
>         bandwidth 3mbit
>
>     # leave 30KB (240kbps) to other machines in the network
>     tc class add dev wlan0 parent 1: classid 1:1 cbq      \
>         rate 2832kbit allot 1500 prio 5 bounded isolated
>
>     # redirect all traffic on 192.168.0.5 to the previous class
>     tc filter add dev wlan0 parent 1: protocol ip prio 16 \
>         u32 match ip dst 192.168.0.5 flowid 1:1
>
>     # change the hashing algorithm every 10s to avoid collisions
>     tc qdisc add dev wlan0 parent 1:1 sfq perturb 10
> fi
>
> The problem is that I have tried setting 2832kbit to very small values
> for testing (like 16kbit), but I still can browse the web at high speed.
> The problem is not in NetworkManager, because I'm testing the script
> manually. Also, tc doesn't return any errors.
>
> I have found that by changing dst 192.168.0.5 to src 192.168.0.5, the
> upload speed is reliably limited, but I still haven't figured how to get
> the download speed to work, which is the most important for me.
>
> The following approach I have taken from wondershaper:
>
> tc qdisc  add dev wlan0 handle ffff: ingress
> tc filter add dev wlan0 parent ffff: protocol ip prio 50 u32 match ip \
>     dst 192.168.0.5 police rate 200kbps burst 10k drop flowid :1
>
> But the problem with this one is that it is very unreliable, it results in
> unstable download speeds (if set to 200, it downloads at ~80-120).
>
> Thank you!
> Teresa and Junior
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Mike SCHMIDT
CTO
Intello Technologies Inc.
mike.schmidt@intello.com
Canada: 1-888-404-6261 x320
USA: 1-888-404-6268 x320
Mobile: 514-409-6898
www.intello.com

Re: Can't get tc to limit network traffic

[Teresa e Junior]

On Mon, 15 Sep 2014 00:18:43 -0400, Mike Schmidt wrote:
> Basically, you cannot control inbound traffic with tc, except through
> policing, which is what the wondershaper rule does. Policing does not
> shape traffic, it simply drops the excess, which is why it is so
> unreliable.  That is why the throttling works when only you set the
> src to 192.168.0.5. The solution is to use an intermediate device,
> like ifb, or imq, depending on the distribution you are running.  A
> typical example is here:
> http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb

Thank you so much for your answer, Mike!

I had a look at the link you gave to me, and after wrestling against
some errors, I came up with this:

#!/bin/sh

tc qdisc del dev ifb0  root
tc qdisc del dev wlan0 ingress

modprobe ifb

tc qdisc  add dev ifb0 root handle 1: prio
tc qdisc  add dev ifb0 parent 1:1 handle 10: sfq
tc qdisc  add dev ifb0 parent 1:2 handle 20: tbf rate 20kbit \
   buffer 1600 limit 3000
tc qdisc  add dev ifb0 parent 1:3 handle 30: sfq
tc filter add dev ifb0 protocol ip pref 1 parent 1: handle 1 fw classid 1:1
tc filter add dev ifb0 protocol ip pref 2 parent 1: handle 2 fw classid 1:2

ifconfig ifb0 up

tc qdisc  add dev wlan0 ingress
tc filter add dev wlan0 parent ffff: protocol ip prio 10 u32 \
   match u32 0 0 flowid 1:1       \
   action xt -j MARK --set-mark 1 \
   action mirred egress redirect dev ifb0

I'm using xt instead of ipt because ipt gives me the error "bad action
type ipt" and, as mentioned in Debian bug #710450, ipt is just a symlink
to xt:
$ readlink /usr/lib/tc/m_ipt.so
m_xt.so

The problem with this approach is that it doesn't work, actually, but
probably because I'm doing something wrong. I suppose download and
upload would have to be limited to 20kbit, right? When I run the above
script, the speeds are not limited at all. I get only:

tablename: mangle hook: NF_IP_PRE_ROUTING
  	target:  MARK set 0x1 index 0

Re: Can't get tc to limit network traffic

[Mike Schmidt]

I don't know whether the iptable filter works in this circumstance, we
filter by ip address, more like this:

/sbin/tc filter add dev ifb0 parent 1:0 protocol ip prio 100 u32 match
ip src 172.20.0.0/20 classid 1:200

You need to use "tc -s filter show dev ifb0" to see if there is
traffic going through any of the rules, and you can then try to debug
that rule.
There have been several mentions about problems with iptables marks
not being seen by tc, except this depends on the kernel.

Oh, also note that ifb's ideas of what is source and destination are
the opposite of br0, so you might try both ways to begin. Once you see
traffic flowing in the filters, you can adjust your speed accordingly.

On Mon, Sep 15, 2014 at 1:09 AM, Teresa e Junior
<teresaejunior@gmail.com> wrote:
> On Mon, 15 Sep 2014 00:18:43 -0400, Mike Schmidt wrote:
>>
>> Basically, you cannot control inbound traffic with tc, except through
>> policing, which is what the wondershaper rule does. Policing does not
>> shape traffic, it simply drops the excess, which is why it is so
>> unreliable.  That is why the throttling works when only you set the
>> src to 192.168.0.5. The solution is to use an intermediate device,
>> like ifb, or imq, depending on the distribution you are running.  A
>> typical example is here:
>> http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb
>
>
> Thank you so much for your answer, Mike!
>
> I had a look at the link you gave to me, and after wrestling against
> some errors, I came up with this:
>
> #!/bin/sh
>
> tc qdisc del dev ifb0  root
> tc qdisc del dev wlan0 ingress
>
> modprobe ifb
>
> tc qdisc  add dev ifb0 root handle 1: prio
> tc qdisc  add dev ifb0 parent 1:1 handle 10: sfq
> tc qdisc  add dev ifb0 parent 1:2 handle 20: tbf rate 20kbit \
>   buffer 1600 limit 3000
> tc qdisc  add dev ifb0 parent 1:3 handle 30: sfq
> tc filter add dev ifb0 protocol ip pref 1 parent 1: handle 1 fw classid 1:1
> tc filter add dev ifb0 protocol ip pref 2 parent 1: handle 2 fw classid 1:2
>
> ifconfig ifb0 up
>
> tc qdisc  add dev wlan0 ingress
> tc filter add dev wlan0 parent ffff: protocol ip prio 10 u32 \
>   match u32 0 0 flowid 1:1       \
>   action xt -j MARK --set-mark 1 \
>   action mirred egress redirect dev ifb0
>
> I'm using xt instead of ipt because ipt gives me the error "bad action
> type ipt" and, as mentioned in Debian bug #710450, ipt is just a symlink
> to xt:
> $ readlink /usr/lib/tc/m_ipt.so
> m_xt.so
>
> The problem with this approach is that it doesn't work, actually, but
> probably because I'm doing something wrong. I suppose download and
> upload would have to be limited to 20kbit, right? When I run the above
> script, the speeds are not limited at all. I get only:
>
> tablename: mangle hook: NF_IP_PRE_ROUTING
>         target:  MARK set 0x1 index 0



-- 
Mike SCHMIDT
CTO
Intello Technologies Inc.
mike.schmidt@intello.com
Canada: 1-888-404-6261 x320
USA: 1-888-404-6268 x320
Mobile: 514-409-6898
www.intello.com

Re: Can't get tc to limit network traffic

[Andy Furniss]

Teresa e Junior wrote:

> #!/bin/sh
>
> tc qdisc del dev ifb0  root
> tc qdisc del dev wlan0 ingress
>
> modprobe ifb
>
> tc qdisc  add dev ifb0 root handle 1: prio
> tc qdisc  add dev ifb0 parent 1:1 handle 10: sfq
> tc qdisc  add dev ifb0 parent 1:2 handle 20: tbf rate 20kbit \
>    buffer 1600 limit 3000
> tc qdisc  add dev ifb0 parent 1:3 handle 30: sfq
> tc filter add dev ifb0 protocol ip pref 1 parent 1: handle 1 fw classid 1:1
> tc filter add dev ifb0 protocol ip pref 2 parent 1: handle 2 fw classid 1:2

Well handle 2 would need iptables mark 2 - and I don't think using 
iptables is needed anyway.

Using tc to set it, historically at least, has been a bit hit and miss 
requiring matching versions of things.

TBH this all seems over complicated if all you want to do is send all 
ipv4 to ifb0 and limit to a couple of meg.

Adding a child of sfq or fq_codel will help stop bulk flows blocking eg dns.

I must admit I made up the params for fq_codel below - maybe someone 
will suggest better for 2mbit rate.

tc qdisc add dev ifb0 root handle 1:0 htb default 1
tc class add dev ifb0 parent 1:0  classid 1:1 htb rate 2mbit
tc qdisc add dev ifb0 parent 1:1 handle 10: fq_codel noecn target 50ms 
interval 100ms

I would just delete the action xt line from below.

> tc qdisc  add dev wlan0 ingress
> tc filter add dev wlan0 parent ffff: protocol ip prio 10 u32 \
>    match u32 0 0 flowid 1:1       \
>    action xt -j MARK --set-mark 1 \
>    action mirred egress redirect dev ifb0

More generally I am guessing you are limiting this box to stop it 
hogging the wan bandwidth rather than wireless lan.

If the download speed of the wan is not much higher than what you set 
you may need to set lower.

Wherever/however you shape incoming from wan you need to back off as you 
are at the wrong end of the bottleneck.

How much you have to back off depends on how hard you hit the connection 
eg if your box is downloading one tcp it will work closer to the limit 
than 100 tcps spread out all over the place like bittorrent.

Re: Can't get tc to limit network traffic

[Dave Taht]

On Mon, Sep 15, 2014 at 7:03 PM, Andy Furniss <adf.lists@gmail.com> wrote:
> Teresa e Junior wrote:
>
>> #!/bin/sh
>>
>> tc qdisc del dev ifb0  root
>> tc qdisc del dev wlan0 ingress
>>
>> modprobe ifb
>>
>> tc qdisc  add dev ifb0 root handle 1: prio
>> tc qdisc  add dev ifb0 parent 1:1 handle 10: sfq
>> tc qdisc  add dev ifb0 parent 1:2 handle 20: tbf rate 20kbit \
>>    buffer 1600 limit 3000
>> tc qdisc  add dev ifb0 parent 1:3 handle 30: sfq
>> tc filter add dev ifb0 protocol ip pref 1 parent 1: handle 1 fw classid
>> 1:1
>> tc filter add dev ifb0 protocol ip pref 2 parent 1: handle 2 fw classid
>> 1:2
>
>
> Well handle 2 would need iptables mark 2 - and I don't think using iptables
> is needed anyway.
>
> Using tc to set it, historically at least, has been a bit hit and miss
> requiring matching versions of things.
>
> TBH this all seems over complicated if all you want to do is send all ipv4
> to ifb0 and limit to a couple of meg.
>
> Adding a child of sfq or fq_codel will help stop bulk flows blocking eg dns.
>
> I must admit I made up the params for fq_codel below - maybe someone will
> suggest better for 2mbit rate.
>
> tc qdisc add dev ifb0 root handle 1:0 htb default 1
> tc class add dev ifb0 parent 1:0  classid 1:1 htb rate 2mbit
> tc qdisc add dev ifb0 parent 1:1 handle 10: fq_codel noecn target 50ms
> interval 100ms

the fq_codel internet draft[1] suggests that "interval"

SHOULD be set on the order of the worst-case RTT
   through the bottleneck to give end-points sufficient time to react.

so if you are using a satellite comms system, use 1200ms as the interval,
if you are in new zealand, use 240ms, for most of the rest of the world, 100ms
is the default and "good enough".

As for "interval"

The default target value is 5 ms, but this value SHOULD be tuned to
   be at least the transmission time of a single MTU-sized packet at the
   prevalent egress link speed (which for e.g. 1Mbps and MTU 1500 is
   ~15ms).

and should be 5-10% of the interval, regardless. We hope to remove the
target parameter entirely in some future version of codel - it's actually
the interaction with tbf/htb/hfsc which buffers up an extra packet
 here that requires it be increased at very low bandwidths.

So in this case, 2mbit, a target 8ms would be sufficient. It doesn't
hurt to go bigger, but 50ms is overmuch.

As for ecn, there is generally little harm in leaving it on, and some
benefit, particularly at higher rates and longer RTTs, but
you have to ecn enable the rest of your network to suit [2]

[1] http://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00#section-4.1.1
[2] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf


> I would just delete the action xt line from below.
>
>> tc qdisc  add dev wlan0 ingress
>> tc filter add dev wlan0 parent ffff: protocol ip prio 10 u32 \
>>    match u32 0 0 flowid 1:1       \
>>    action xt -j MARK --set-mark 1 \
>>    action mirred egress redirect dev ifb0
>
>
> More generally I am guessing you are limiting this box to stop it hogging
> the wan bandwidth rather than wireless lan.
>
> If the download speed of the wan is not much higher than what you set you
> may need to set lower.
>
> Wherever/however you shape incoming from wan you need to back off as you are
> at the wrong end of the bottleneck.
>
> How much you have to back off depends on how hard you hit the connection eg
> if your box is downloading one tcp it will work closer to the limit than 100
> tcps spread out all over the place like bittorrent.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: Can't get tc to limit network traffic

[Andy Furniss]

Dave Taht wrote:

>> tc qdisc add dev ifb0 parent 1:1 handle 10: fq_codel noecn target
>> 50ms interval 100ms
>
> the fq_codel internet draft[1] suggests that "interval"
>
> SHOULD be set on the order of the worst-case RTT through the
> bottleneck to give end-points sufficient time to react.
>
> so if you are using a satellite comms system, use 1200ms as the
> interval, if you are in new zealand, use 240ms, for most of the rest
> of the world, 100ms is the default and "good enough".
>
> As for "interval"
>
> The default target value is 5 ms, but this value SHOULD be tuned to
> be at least the transmission time of a single MTU-sized packet at
> the prevalent egress link speed (which for e.g. 1Mbps and MTU 1500
> is ~15ms).
>
> and should be 5-10% of the interval, regardless. We hope to remove
> the target parameter entirely in some future version of codel - it's
> actually the interaction with tbf/htb/hfsc which buffers up an extra
> packet here that requires it be increased at very low bandwidths.
>
> So in this case, 2mbit, a target 8ms would be sufficient. It doesn't
> hurt to go bigger, but 50ms is overmuch.

Thanks for that, I am really out of date with the new stuff.

While doing a quick test of this I just noticed that fq_codel saw
maxpacket 4542 but htb didn't count a giant - so I guess htbs limit is
higher than it used to be at least.

Of course the packet was fake (I know nothing that big goes on the
wire), but it serves to remind me that generic-receive-offload is on by
default and if I were to shape for real on this box it would be better
off (for low speed shaping).

ethtool -K eth0 gro off

Re: Can't get tc to limit network traffic

[Dave Taht]

On Mon, Sep 15, 2014 at 8:27 PM, Andy Furniss <adf.lists@gmail.com> wrote:
> Dave Taht wrote:
>
>>> tc qdisc add dev ifb0 parent 1:1 handle 10: fq_codel noecn target
>>> 50ms interval 100ms
>>
>>
>> the fq_codel internet draft[1] suggests that "interval"
>>
>> SHOULD be set on the order of the worst-case RTT through the
>> bottleneck to give end-points sufficient time to react.
>>
>> so if you are using a satellite comms system, use 1200ms as the
>> interval, if you are in new zealand, use 240ms, for most of the rest
>> of the world, 100ms is the default and "good enough".
>>
>> As for "interval"

Thinko: meant "target"

>>
>> The default target value is 5 ms, but this value SHOULD be tuned to
>> be at least the transmission time of a single MTU-sized packet at
>> the prevalent egress link speed (which for e.g. 1Mbps and MTU 1500
>> is ~15ms).
>>
>> and should be 5-10% of the interval, regardless. We hope to remove
>> the target parameter entirely in some future version of codel - it's
>> actually the interaction with tbf/htb/hfsc which buffers up an extra
>> packet here that requires it be increased at very low bandwidths.
>>
>> So in this case, 2mbit, a target 8ms would be sufficient. It doesn't
>> hurt to go bigger, but 50ms is overmuch.
>
>
> Thanks for that, I am really out of date with the new stuff.
>
> While doing a quick test of this I just noticed that fq_codel saw
> maxpacket 4542 but htb didn't count a giant - so I guess htbs limit is
> higher than it used to be at least.
>
> Of course the packet was fake (I know nothing that big goes on the
> wire), but it serves to remind me that generic-receive-offload is on by
> default and if I were to shape for real on this box it would be better
> off (for low speed shaping).
>
> ethtool -K eth0 gro off

Note that on a router GRO should be off on *all* interfaces. In fact I recommend
tso,gso,gro, and lro all be off on a router. Let packets be packets!

There is a debloat.sh script out there that helps. See also:

http://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

If anyone out there is up for patching new kernels and doing some
benchmarking, we have a new all-singing-all-dancing "tbf + diffserv +
fq_codel" single line rate limiting qdisc under development,
tenatively called "cake". If you want a shot at it, let me know.


>
>



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: Can't get tc to limit network traffic

[Teresa e Junior]

Thanks for all the help to everybody! Since all this is a bit too 
complicated for a desktop user, I'll try to understand it better when I 
have more time available. I see there are a lot of man pages for tc-*!

I have tried some of the tips that you have given without much success, 
but from my past experience, I won't get anywhere unless I get to 
understand what each of the commands do, although networking is 
something I've been always scared to death!