Can't get the ingress policer to work

Can't get the ingress policer to work

[Sebastian Arcus]

I'm trying to limit the bandwidth on the ingress leg of my Internet 
connection. I'm using the code from lartc.org:


tc qdisc add dev eth0 handle ffff: ingress

tc filter add dev eth0 parent ffff: protocol ip prio 50 \
     u32 match ip src 0.0.0.0/0 police rate 500kbit \
     burst 10k drop flowid :1


I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k, 
10k, 100k, 1000k. Nothing seems to make any difference. The test 
download starts at about 20Mbytes/second - then keeps on slowing down 
and stalling intermittently all the way down to 6kbytes/second. Then it 
bobs up and down, stalling all the time, between 5kbytes/second and 
17kbytes/second. There seems to be absolutely no relation between the 
rate I set and the resulting download rate.

I'm testing on two VM's. With no traffic shaping on, I get about 36 
megabytes/second clean speed between the two vms. Kernel on both sides 
is 3.8.4. I'm only applying traffic shaping on one of the vm's.

Any suggestions would be much appreciated. I ran out of ideas so far. 
I've reread the tc-htb man page, searched google on the ingress queue - 
but can't see what am I missing.

Re: Can't get the ingress policer to work

[Sebastian Arcus]

Thanks for replying. I am not familiar with the ifb interface. I will 
look it up and see if I can make sense of things.



On 09/05/13 10:18, Faysal Banna wrote:
> Good Day Sir.
>
> Why don't you use ifb and mirred towards ifb0 or ifb1 and thus you can
> rate limit ingress traffic easy.
>
> Much Regards
> *Faysal Banna*
> _Chief Section Surface Observation_
> _Meteorological Department_
> _Civil Aviation_
> _Rafic Harriri International Airport_
> _Beirut Lebanon_
> _Mobile : +961-3-258043_
> _Tel : +961-1-628000 EXT: 2000_
> __
> On 05/09/2013 11:57 AM, Sebastian Arcus wrote:
>> I'm trying to limit the bandwidth on the ingress leg of my Internet
>> connection. I'm using the code from lartc.org:
>>
>>
>> tc qdisc add dev eth0 handle ffff: ingress
>>
>> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>>     u32 match ip src 0.0.0.0/0 police rate 500kbit \
>>     burst 10k drop flowid :1
>>
>>
>> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
>> 10k, 100k, 1000k. Nothing seems to make any difference. The test
>> download starts at about 20Mbytes/second - then keeps on slowing down
>> and stalling intermittently all the way down to 6kbytes/second. Then
>> it bobs up and down, stalling all the time, between 5kbytes/second and
>> 17kbytes/second. There seems to be absolutely no relation between the
>> rate I set and the resulting download rate.
>>
>> I'm testing on two VM's. With no traffic shaping on, I get about 36
>> megabytes/second clean speed between the two vms. Kernel on both sides
>> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>>
>> Any suggestions would be much appreciated. I ran out of ideas so far.
>> I've reread the tc-htb man page, searched google on the ingress queue
>> - but can't see what am I missing.
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


-- 
Linux vehicle CCTV - www.open-t.co.uk/iroko

Re: Can't get the ingress policer to work

[Ian Macintosh]

How are you testing the rate?  If you're using some random protocol
that will cloud the issue.  Try using raw tcp/udp via a test tool.
iperf comes to mind.

On 9 May 2013 09:57, Sebastian Arcus <shop@open-t.co.uk> wrote:
> I'm trying to limit the bandwidth on the ingress leg of my Internet
> connection. I'm using the code from lartc.org:
>
>
> tc qdisc add dev eth0 handle ffff: ingress
>
> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>     u32 match ip src 0.0.0.0/0 police rate 500kbit \
>     burst 10k drop flowid :1
>
>
> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k, 10k,
> 100k, 1000k. Nothing seems to make any difference. The test download starts
> at about 20Mbytes/second - then keeps on slowing down and stalling
> intermittently all the way down to 6kbytes/second. Then it bobs up and down,
> stalling all the time, between 5kbytes/second and 17kbytes/second. There
> seems to be absolutely no relation between the rate I set and the resulting
> download rate.
>
> I'm testing on two VM's. With no traffic shaping on, I get about 36
> megabytes/second clean speed between the two vms. Kernel on both sides is
> 3.8.4. I'm only applying traffic shaping on one of the vm's.
>
> Any suggestions would be much appreciated. I ran out of ideas so far. I've
> reread the tc-htb man page, searched google on the ingress queue - but can't
> see what am I missing.
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Ian Macintosh
Technical Director
KCS Total Solutions Ltd

Hemel Hempstead – Tel: (01442) 251-514
Fax: (01442) 200-198
Web: http://www.kcsts.co.uk/

KCS Total Solutions Ltd, registered in England and Wales. Registered
number 3792344. Registered office: 3 Kensworth Gate, 200-204 High
Street South, Dunstable, LU63HS.

NOTICE: The contents of this email and any attachments are strictly
confidential and intended solely for the attention and use of the
named addressee(s). If you are not the addressee(s) you may not
disclose, copy, distribute, or retain this message without our prior
written authority. If received in error, please contact KCS Total
Solutions Ltd on (01442) 251-514 quoting the name of the sender.

Re: Can't get the ingress policer to work

[Sebastian Arcus]

On 09/05/13 09:57, Sebastian Arcus wrote:
> I'm trying to limit the bandwidth on the ingress leg of my Internet
> connection. I'm using the code from lartc.org:
>
>
> tc qdisc add dev eth0 handle ffff: ingress
>
> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>      u32 match ip src 0.0.0.0/0 police rate 500kbit \
>      burst 10k drop flowid :1
>
>
> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
> 10k, 100k, 1000k. Nothing seems to make any difference. The test
> download starts at about 20Mbytes/second - then keeps on slowing down
> and stalling intermittently all the way down to 6kbytes/second. Then it
> bobs up and down, stalling all the time, between 5kbytes/second and
> 17kbytes/second. There seems to be absolutely no relation between the
> rate I set and the resulting download rate.
>
> I'm testing on two VM's. With no traffic shaping on, I get about 36
> megabytes/second clean speed between the two vms. Kernel on both sides
> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>
> Any suggestions would be much appreciated. I ran out of ideas so far.
> I've reread the tc-htb man page, searched google on the ingress queue -
> but can't see what am I missing.

I have just retried the exact same settings, but on a real server with 
an ADSL connection to the internet (well, actually an ethernet 
connection which goes through an ADSL router). On this server, the 
ingress policing is working perfectly fine. Maybe the KVM vm's or the 
virtio network interface is playing havoc with tc?

The real server has kernel 2.6.38 - while the vm has 3.8.4. Could the 
kernel version be an issue? Have there been changes in traffic 
shaping/policing code in newer kernels? Could there be other settings 
which are interfering with the ingress policing? Maybe the MRU on the 
interface?

Re: Can't get the ingress policer to work

[Andy Furniss]

Sebastian Arcus wrote:
> I'm trying to limit the bandwidth on the ingress leg of my Internet
> connection. I'm using the code from lartc.org:
>
>
> tc qdisc add dev eth0 handle ffff: ingress
>
> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>      u32 match ip src 0.0.0.0/0 police rate 500kbit \
>      burst 10k drop flowid :1
>
>
> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
> 10k, 100k, 1000k. Nothing seems to make any difference. The test
> download starts at about 20Mbytes/second - then keeps on slowing down
> and stalling intermittently all the way down to 6kbytes/second. Then it
> bobs up and down, stalling all the time, between 5kbytes/second and
> 17kbytes/second. There seems to be absolutely no relation between the
> rate I set and the resulting download rate.
>
> I'm testing on two VM's. With no traffic shaping on, I get about 36
> megabytes/second clean speed between the two vms. Kernel on both sides
> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>
> Any suggestions would be much appreciated. I ran out of ideas so far.
> I've reread the tc-htb man page, searched google on the ingress queue -
> but can't see what am I missing.


I've never tested with a virtual machine so don't know exactly what to 
do. The issue is that by default the kernel/driver will aggregate 
inbound packets into larger ones and this doesn't work well with 
policers default mtu setting as it drops them.

I have suggested mtu 3100 in the past as an extra parameter as it 
happened to work on a quick test on a real 100m nic - maybe for vm you 
need to go higher, or just turn off any sort of offload for the virtual 
device with ethtool and see if that helps.

ethtool -k <devname>

to see what the settings are (generic-receive-offload is likely the 
culprit) and eg.

ethtool -K <devname> gro off

Re: Can't get the ingress policer to work

[fanfei]

于 2013/5/9 18:09, Sebastian Arcus 写道:
> On 09/05/13 09:57, Sebastian Arcus wrote:
>> I'm trying to limit the bandwidth on the ingress leg of my Internet
>> connection. I'm using the code from lartc.org:
>>
>>
>> tc qdisc add dev eth0 handle ffff: ingress
>>
>> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>> u32 match ip src 0.0.0.0/0 police rate 500kbit \
>> burst 10k drop flowid :1
>>
>>
>> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
>> 10k, 100k, 1000k. Nothing seems to make any difference. The test
>> download starts at about 20Mbytes/second - then keeps on slowing down
>> and stalling intermittently all the way down to 6kbytes/second. Then it
>> bobs up and down, stalling all the time, between 5kbytes/second and
>> 17kbytes/second. There seems to be absolutely no relation between the
>> rate I set and the resulting download rate.
>>
>> I'm testing on two VM's. With no traffic shaping on, I get about 36
>> megabytes/second clean speed between the two vms. Kernel on both sides
>> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>>
>> Any suggestions would be much appreciated. I ran out of ideas so far.
>> I've reread the tc-htb man page, searched google on the ingress queue -
>> but can't see what am I missing.
>
> I have just retried the exact same settings, but on a real server with 
> an ADSL connection to the internet (well, actually an ethernet 
> connection which goes through an ADSL router). On this server, the 
> ingress policing is working perfectly fine. Maybe the KVM vm's or the 
> virtio network interface is playing havoc with tc?
>
> The real server has kernel 2.6.38 - while the vm has 3.8.4. Could the 
> kernel version be an issue? Have there been changes in traffic 
> shaping/policing code in newer kernels? Could there be other settings 
> which are interfering with the ingress policing? Maybe the MRU on the 
> interface?
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

try set mtu, like
tc filter add dev eth0 parent ffff: protocol ip prio 50 \
u32 match ip src 0.0.0.0/0 police rate 500kbit \
burst 10k mtu 100kb drop flowid :1

100kb works for me, maybe you should set another value

Re: Can't get the ingress policer to work

[Sebastian Arcus]

I was just using an scp command. I figured as, with ingress, everything 
goes into a single queue - it should show fairly relevant results - even 
if it wasn't down to the tiniest fraction of precision.



On 09/05/13 10:51, Ian Macintosh wrote:
> How are you testing the rate?  If you're using some random protocol
> that will cloud the issue.  Try using raw tcp/udp via a test tool.
> iperf comes to mind.
>
> On 9 May 2013 09:57, Sebastian Arcus <shop@open-t.co.uk> wrote:
>> I'm trying to limit the bandwidth on the ingress leg of my Internet
>> connection. I'm using the code from lartc.org:
>>
>>
>> tc qdisc add dev eth0 handle ffff: ingress
>>
>> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>>      u32 match ip src 0.0.0.0/0 police rate 500kbit \
>>      burst 10k drop flowid :1
>>
>>
>> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k, 10k,
>> 100k, 1000k. Nothing seems to make any difference. The test download starts
>> at about 20Mbytes/second - then keeps on slowing down and stalling
>> intermittently all the way down to 6kbytes/second. Then it bobs up and down,
>> stalling all the time, between 5kbytes/second and 17kbytes/second. There
>> seems to be absolutely no relation between the rate I set and the resulting
>> download rate.
>>
>> I'm testing on two VM's. With no traffic shaping on, I get about 36
>> megabytes/second clean speed between the two vms. Kernel on both sides is
>> 3.8.4. I'm only applying traffic shaping on one of the vm's.
>>
>> Any suggestions would be much appreciated. I ran out of ideas so far. I've
>> reread the tc-htb man page, searched google on the ingress queue - but can't
>> see what am I missing.
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>


-- 
Linux vehicle CCTV - www.open-t.co.uk/iroko

Re: Can't get the ingress policer to work

[Sebastian Arcus]

On 09/05/13 11:28, fanfei wrote:
> 于 2013/5/9 18:09, Sebastian Arcus 写道:
>> On 09/05/13 09:57, Sebastian Arcus wrote:
>>> I'm trying to limit the bandwidth on the ingress leg of my Internet
>>> connection. I'm using the code from lartc.org:
>>>
>>>
>>> tc qdisc add dev eth0 handle ffff: ingress
>>>
>>> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>>> u32 match ip src 0.0.0.0/0 police rate 500kbit \
>>> burst 10k drop flowid :1
>>>
>>>
>>> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
>>> 10k, 100k, 1000k. Nothing seems to make any difference. The test
>>> download starts at about 20Mbytes/second - then keeps on slowing down
>>> and stalling intermittently all the way down to 6kbytes/second. Then it
>>> bobs up and down, stalling all the time, between 5kbytes/second and
>>> 17kbytes/second. There seems to be absolutely no relation between the
>>> rate I set and the resulting download rate.
>>>
>>> I'm testing on two VM's. With no traffic shaping on, I get about 36
>>> megabytes/second clean speed between the two vms. Kernel on both sides
>>> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>>>
>>> Any suggestions would be much appreciated. I ran out of ideas so far.
>>> I've reread the tc-htb man page, searched google on the ingress queue -
>>> but can't see what am I missing.
>>
>> I have just retried the exact same settings, but on a real server with
>> an ADSL connection to the internet (well, actually an ethernet
>> connection which goes through an ADSL router). On this server, the
>> ingress policing is working perfectly fine. Maybe the KVM vm's or the
>> virtio network interface is playing havoc with tc?
>>
>> The real server has kernel 2.6.38 - while the vm has 3.8.4. Could the
>> kernel version be an issue? Have there been changes in traffic
>> shaping/policing code in newer kernels? Could there be other settings
>> which are interfering with the ingress policing? Maybe the MRU on the
>> interface?
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> try set mtu, like
> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
> u32 match ip src 0.0.0.0/0 police rate 500kbit \
> burst 10k mtu 100kb drop flowid :1
>
> 100kb works for me, maybe you should set another value

Considering the ingress works on the downlink - shouldn't it be affected 
by the MRU - not the MTU? I could be wrong - I just always thought the 
MTU affects the size of the packets going out

Re: Can't get the ingress policer to work

[Andy Furniss]

Sebastian Arcus wrote:
> I'm trying to limit the bandwidth on the ingress leg of my Internet
> connection. I'm using the code from lartc.org:
>
>
> tc qdisc add dev eth0 handle ffff: ingress
>
> tc filter add dev eth0 parent ffff: protocol ip prio 50 \
>      u32 match ip src 0.0.0.0/0 police rate 500kbit \
>      burst 10k drop flowid :1
>
>
> I've tried rates of 1Mbit, 10Mbit, 100Mbit. I've tried bursts of 1k,
> 10k, 100k, 1000k. Nothing seems to make any difference. The test
> download starts at about 20Mbytes/second - then keeps on slowing down
> and stalling intermittently all the way down to 6kbytes/second. Then it
> bobs up and down, stalling all the time, between 5kbytes/second and
> 17kbytes/second. There seems to be absolutely no relation between the
> rate I set and the resulting download rate.
>
> I'm testing on two VM's. With no traffic shaping on, I get about 36
> megabytes/second clean speed between the two vms. Kernel on both sides
> is 3.8.4. I'm only applying traffic shaping on one of the vm's.
>
> Any suggestions would be much appreciated. I ran out of ideas so far.
> I've reread the tc-htb man page, searched google on the ingress queue -
> but can't see what am I missing.


I've never tested with a virtual machine so don't know exactly what to 
do. The issue is that by default the kernel/driver will aggregate 
inbound packets into larger ones and this doesn't work well with 
policers default mtu setting as it drops them.

I have suggested mtu 3100 in the past as an extra parameter as it 
happened to work on a quick test on a real 100m nic - maybe for vm you 
need to go higher, or just turn off any sort of offload for the virtual 
device with ethtool and see if that helps.

ethtool -k <devname>

to see what the settings are (generic-receive-offload is likely the 
culprit) and eg.

ethtool -K <devname> gro off