prio queue not working as expected?

prio queue not working as expected?

[Wolfgang Hennerbichler]

hi tc folks, 

I'm trying to set up a bridged traffic shaper, and I am experimenting with the prio qdisc. 
I've take the example from www.lartc.org: 

tc qdisc add dev wan0 root handle 1: prio 
tc qdisc add dev wan0 parent 1:1 handle 10: sfq
tc qdisc add dev wan0 parent 1:2 handle 20: tbf rate 20kbit buffer 1600 limit 3000
tc qdisc add dev wan0 parent 1:3 handle 30: sfq             

unfortunately, it seems as if it doesn't detect the TOS of my scp traffic. when I do a scp from a box behind the bridge, only qdisc 20: increases it's counters, I can't create counter values in 10: or 30: by issuing telnet or ssh commands or whatever. 

tc -s qdisc show dev wan0 | grep qdisc -A1
qdisc prio 1: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 69328 bytes 457 pkt (dropped 0, overlimits 0 requeues 0) 
--
qdisc sfq 10: parent 1:1 limit 127p quantum 1514b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
--
qdisc tbf 20: parent 1:2 rate 20000bit burst 1600b lat 560.0ms 
 Sent 69328 bytes 457 pkt (dropped 0, overlimits 76 requeues 0) 
--
qdisc sfq 30: parent 1:3 limit 127p quantum 1514b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 

The system is running within a VM on linux kernel 3.7.10. 
I can't figure out what's wrong here. Any help is really appreciated. 

Wolfgang
-- 
http://www.wogri.at

Re: prio queue not working as expected?

[Andrew Beverley]

On Wed, 2013-06-19 at 15:15 +0200, Wolfgang Hennerbichler wrote:
> hi tc folks, 
> 
> I'm trying to set up a bridged traffic shaper, and I am experimenting with the prio qdisc. 
> I've take the example from www.lartc.org: 
> 
> tc qdisc add dev wan0 root handle 1: prio 
> tc qdisc add dev wan0 parent 1:1 handle 10: sfq
> tc qdisc add dev wan0 parent 1:2 handle 20: tbf rate 20kbit buffer 1600 limit 3000
> tc qdisc add dev wan0 parent 1:3 handle 30: sfq             
> 
> unfortunately, it seems as if it doesn't detect the TOS of my scp
> traffic. when I do a scp from a box behind the bridge, only qdisc 20:
> increases it's counters,

For an scp command, it should be the 20: or 30: counters that increase,
as it's bulk traffic.

> I can't create counter values in 10: or 30: by issuing telnet or ssh
> commands or whatever. 

But normal ssh commands should, of course, be in 10:

I don't know why it is not working for you, but I've just tried the
above commands myself and it works for me, so at least you know that the
example is correct.

> The system is running within a VM on linux kernel 3.7.10. 
> I can't figure out what's wrong here. Any help is really appreciated. 

My test was with Linux 3.0.0, not inside a VM.

You could maybe use some iptables logging to see if the packets have the
TOS value that you expect? You could also try setting some TOS values
with iptables and see if that works.

Andy

Re: prio queue not working as expected?

[Wolfgang Hennerbichler]

On Jun 20, 2013, at 09:25 , Andrew Beverley <andy@andybev.com> wrote:
> 
>> I can't create counter values in 10: or 30: by issuing telnet or ssh
>> commands or whatever. 
> 
> But normal ssh commands should, of course, be in 10:
> 
> I don't know why it is not working for you, but I've just tried the
> above commands myself and it works for me, so at least you know that the
> example is correct.

ok, thank you. I still can't figure out why this happens, but at least I know the documentation works. 

>> The system is running within a VM on linux kernel 3.7.10. 
>> I can't figure out what's wrong here. Any help is really appreciated. 
> 
> My test was with Linux 3.0.0, not inside a VM.
> 
> You could maybe use some iptables logging to see if the packets have the
> TOS value that you expect? You could also try setting some TOS values
> with iptables and see if that works.

TOS values are correct (tcpdump -vv). If I enqeue packets into the first PRIO class with a filter, these packets are priorized (tested with icmp echo replies + scp transfer), but still the counters don't increase. Seems to be a problem with displaying the correct counter values. 

> Andy

Wolfgang

> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: prio queue not working as expected?

[Wolfgang Hennerbichler]

-- 
http://www.wogri.at

On Jun 21, 2013, at 05:11 , Remy Mudingay <remy.mudingay@gmail.com> wrote:

> Hi Wolfgang,
> 
> Can you post the output from 'tc -s class dev wan0' and also include
> all filters for matching scp, ssh and telnet.

Hi Remy, 

tc -s class show dev wan0
class prio 1:1 parent 1: leaf 10: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:2 parent 1: leaf 20: 
 Sent 97189 bytes 678 pkt (dropped 11, overlimits 409 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:3 parent 1: leaf 30: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class tbf 20:1 parent 20: 

(this is after doing some fiddling with an interactive ssh session). 
There are no filters (neither tc nor iptables mangles), as far as I've understood prio should priorize according to it's priomap. 

here's a tcpdump: 

05:39:28.048426 IP (tos 0x10, ttl 64, id 60303, offset 0, flags [DF], proto TCP (6), length 52)
    x.x.x.x.50356 > y.y.y.y.22: Flags [.], cksum 0x545e (incorrect -> 0xc7ca), seq 0, ack 433, win 653, options [nop,nop,TS val 31327537 ecr 1639911627], length 0

oh. and maybe this is my issue: i see an incorrect checksum here on all those packets. maybe this is causing the troubles. is there a chance that this is happening because my interface is bridged? hm. I wonder how to deal with this (the actual communication works, btw). 

Wolfgang

> Cheers,
> 
> Remy

Re: prio queue not working as expected?

[Wolfgang Hennerbichler]

On Jun 21, 2013, at 11:03 , Remy Mudingay <remy.mudingay@gmail.com> wrote:

> Thanks for providing more info.
> Firstly, the checksum error stems from tcp offloading on the local interface (wan0).

Thank you. I didn't know about TOE and it's implication before. 

> However, the prio qdisc does require the addition of filters when attaching qdisc's to prio classes. Take a look at the following  example :
> 
> http://d3s.mff.cuni.cz/~ceres/sch/osy/text/ch06s02s04.html

wow. Did I misunderstand the prio-qdisc? 
from man tc-prio

CLASSIFICATION
       Three methods are available to PRIO to determine in which band a packet will be enqueued.
…
       with the priomap
              Based on the packet priority, which in turn is derived from the Type of Service assigned to the packet.

              The priomap maps the priority of a packet to a class. The priority can either be  set  directly  from  userspace,  or  be
              derived from the Type of Service of the packet.

              Determines  how  packet priorities, as assigned by the kernel, map to bands. Mapping occurs based on the TOS octet of the
              packet, which looks like this:
… 

to me it seems to be the case that the prio-qdisc would be intelligent enouth to priorize for me. 


> Remy
> On Jun 21, 2013, at 7:42, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
> 
>> 
>> -- 
>> http://www.wogri.at
>> 
>> On Jun 21, 2013, at 05:11 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>> 
>>> Hi Wolfgang,
>>> 
>>> Can you post the output from 'tc -s class dev wan0' and also include
>>> all filters for matching scp, ssh and telnet.
>> 
>> Hi Remy, 
>> 
>> tc -s class show dev wan0
>> class prio 1:1 parent 1: leaf 10: 
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>> backlog 0b 0p requeues 0 
>> class prio 1:2 parent 1: leaf 20: 
>> Sent 97189 bytes 678 pkt (dropped 11, overlimits 409 requeues 0) 
>> backlog 0b 0p requeues 0 
>> class prio 1:3 parent 1: leaf 30: 
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>> backlog 0b 0p requeues 0 
>> class tbf 20:1 parent 20: 
>> 
>> (this is after doing some fiddling with an interactive ssh session). 
>> There are no filters (neither tc nor iptables mangles), as far as I've understood prio should priorize according to it's priomap. 
>> 
>> here's a tcpdump: 
>> 
>> 05:39:28.048426 IP (tos 0x10, ttl 64, id 60303, offset 0, flags [DF], proto TCP (6), length 52)
>>    x.x.x.x.50356 > y.y.y.y.22: Flags [.], cksum 0x545e (incorrect -> 0xc7ca), seq 0, ack 433, win 653, options [nop,nop,TS val 31327537 ecr 1639911627], length 0
>> 
>> oh. and maybe this is my issue: i see an incorrect checksum here on all those packets. maybe this is causing the troubles. is there a chance that this is happening because my interface is bridged? hm. I wonder how to deal with this (the actual communication works, btw). 
>> 
>> Wolfgang
>> 
>>> Cheers,
>>> 
>>> Remy
>> 

Re: prio queue not working as expected?

[Remy Mudingay]

All correct except that you need to classify from the prio class to the leaf qdisc, which in your case is tbf.
I strongly recommend testing this further. 
1. Create the prio qdisc with priomap but without attaching leaf qdisc's to the prio classes. 
2. Then test again by attaching qdiscs to your prio class.
3. Finally test using your current setup but include classification rules.  

Test 1 and 3 will both work.

Remy.


Sent from my iPhone 

On Jun 21, 2013, at 11:09, Wolfgang Hennerbichler <wogri@wogri.com> wrote:

> On Jun 21, 2013, at 11:03 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
> 
>> Thanks for providing more info.
>> Firstly, the checksum error stems from tcp offloading on the local interface (wan0).
> 
> Thank you. I didn't know about TOE and it's implication before. 
> 
>> However, the prio qdisc does require the addition of filters when attaching qdisc's to prio classes. Take a look at the following  example :
>> 
>> http://d3s.mff.cuni.cz/~ceres/sch/osy/text/ch06s02s04.html
> 
> wow. Did I misunderstand the prio-qdisc? 
> from man tc-prio
> 
> CLASSIFICATION
>       Three methods are available to PRIO to determine in which band a packet will be enqueued.
> …
>       with the priomap
>              Based on the packet priority, which in turn is derived from the Type of Service assigned to the packet.
> 
>              The priomap maps the priority of a packet to a class. The priority can either be  set  directly  from  userspace,  or  be
>              derived from the Type of Service of the packet.
> 
>              Determines  how  packet priorities, as assigned by the kernel, map to bands. Mapping occurs based on the TOS octet of the
>              packet, which looks like this:
> … 
> 
> to me it seems to be the case that the prio-qdisc would be intelligent enouth to priorize for me. 
> 
> 
>> Remy
>> On Jun 21, 2013, at 7:42, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
>> 
>>> 
>>> -- 
>>> http://www.wogri.at
>>> 
>>> On Jun 21, 2013, at 05:11 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>>> 
>>>> Hi Wolfgang,
>>>> 
>>>> Can you post the output from 'tc -s class dev wan0' and also include
>>>> all filters for matching scp, ssh and telnet.
>>> 
>>> Hi Remy, 
>>> 
>>> tc -s class show dev wan0
>>> class prio 1:1 parent 1: leaf 10: 
>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>> backlog 0b 0p requeues 0 
>>> class prio 1:2 parent 1: leaf 20: 
>>> Sent 97189 bytes 678 pkt (dropped 11, overlimits 409 requeues 0) 
>>> backlog 0b 0p requeues 0 
>>> class prio 1:3 parent 1: leaf 30: 
>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>> backlog 0b 0p requeues 0 
>>> class tbf 20:1 parent 20: 
>>> 
>>> (this is after doing some fiddling with an interactive ssh session). 
>>> There are no filters (neither tc nor iptables mangles), as far as I've understood prio should priorize according to it's priomap. 
>>> 
>>> here's a tcpdump: 
>>> 
>>> 05:39:28.048426 IP (tos 0x10, ttl 64, id 60303, offset 0, flags [DF], proto TCP (6), length 52)
>>>   x.x.x.x.50356 > y.y.y.y.22: Flags [.], cksum 0x545e (incorrect -> 0xc7ca), seq 0, ack 433, win 653, options [nop,nop,TS val 31327537 ecr 1639911627], length 0
>>> 
>>> oh. and maybe this is my issue: i see an incorrect checksum here on all those packets. maybe this is causing the troubles. is there a chance that this is happening because my interface is bridged? hm. I wonder how to deal with this (the actual communication works, btw). 
>>> 
>>> Wolfgang
>>> 
>>>> Cheers,
>>>> 
>>>> Remy
> 

Re: prio queue not working as expected?

[Wolfgang Hennerbichler]

On 06/21/2013 01:46 PM, Remy Mudingay wrote:
> All correct except that you need to classify from the prio class to the leaf qdisc, which in your case is tbf.
> I strongly recommend testing this further. 

I did test it now. It seems to be a discrepancy between my kernel and my
userland tools.

I tested it on a debian stable stock kernel, now. and things behave as
they should.

> 1. Create the prio qdisc with priomap but without attaching leaf qdisc's to the prio classes. 

I see differentiation.

> 2. Then test again by attaching qdiscs to your prio class.

I see differentiation, also with qdiscs in my prio classes, and even
without classification rules.

> 3. Finally test using your current setup but include classification rules.  

Don't matter anymore. I know that I've been fooled by tc :)

> Test 1 and 3 will both work.
> 
> Remy.

Even Test 2 worked.
Thank you for your help.

Wolfgang

> 
> Sent from my iPhone 
> 
> On Jun 21, 2013, at 11:09, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
> 
>> On Jun 21, 2013, at 11:03 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>>
>>> Thanks for providing more info.
>>> Firstly, the checksum error stems from tcp offloading on the local interface (wan0).
>>
>> Thank you. I didn't know about TOE and it's implication before. 
>>
>>> However, the prio qdisc does require the addition of filters when attaching qdisc's to prio classes. Take a look at the following  example :
>>>
>>> http://d3s.mff.cuni.cz/~ceres/sch/osy/text/ch06s02s04.html
>>
>> wow. Did I misunderstand the prio-qdisc? 
>> from man tc-prio
>>
>> CLASSIFICATION
>>       Three methods are available to PRIO to determine in which band a packet will be enqueued.
>> …
>>       with the priomap
>>              Based on the packet priority, which in turn is derived from the Type of Service assigned to the packet.
>>
>>              The priomap maps the priority of a packet to a class. The priority can either be  set  directly  from  userspace,  or  be
>>              derived from the Type of Service of the packet.
>>
>>              Determines  how  packet priorities, as assigned by the kernel, map to bands. Mapping occurs based on the TOS octet of the
>>              packet, which looks like this:
>> … 
>>
>> to me it seems to be the case that the prio-qdisc would be intelligent enouth to priorize for me. 
>>
>>
>>> Remy
>>> On Jun 21, 2013, at 7:42, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
>>>
>>>>
>>>> -- 
>>>> http://www.wogri.at
>>>>
>>>> On Jun 21, 2013, at 05:11 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>>>>
>>>>> Hi Wolfgang,
>>>>>
>>>>> Can you post the output from 'tc -s class dev wan0' and also include
>>>>> all filters for matching scp, ssh and telnet.
>>>>
>>>> Hi Remy, 
>>>>
>>>> tc -s class show dev wan0
>>>> class prio 1:1 parent 1: leaf 10: 
>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>>> backlog 0b 0p requeues 0 
>>>> class prio 1:2 parent 1: leaf 20: 
>>>> Sent 97189 bytes 678 pkt (dropped 11, overlimits 409 requeues 0) 
>>>> backlog 0b 0p requeues 0 
>>>> class prio 1:3 parent 1: leaf 30: 
>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>>> backlog 0b 0p requeues 0 
>>>> class tbf 20:1 parent 20: 
>>>>
>>>> (this is after doing some fiddling with an interactive ssh session). 
>>>> There are no filters (neither tc nor iptables mangles), as far as I've understood prio should priorize according to it's priomap. 
>>>>
>>>> here's a tcpdump: 
>>>>
>>>> 05:39:28.048426 IP (tos 0x10, ttl 64, id 60303, offset 0, flags [DF], proto TCP (6), length 52)
>>>>   x.x.x.x.50356 > y.y.y.y.22: Flags [.], cksum 0x545e (incorrect -> 0xc7ca), seq 0, ack 433, win 653, options [nop,nop,TS val 31327537 ecr 1639911627], length 0
>>>>
>>>> oh. and maybe this is my issue: i see an incorrect checksum here on all those packets. maybe this is causing the troubles. is there a chance that this is happening because my interface is bridged? hm. I wonder how to deal with this (the actual communication works, btw). 
>>>>
>>>> Wolfgang
>>>>
>>>>> Cheers,
>>>>>
>>>>> Remy
>>


-- 
http://www.wogri.com

Re: prio queue not working as expected?

[Remy Mudingay]

Hi Wolfgang,

Thanks for providing feedback.

Remy

On Jun 24, 2013, at 9:47, Wolfgang Hennerbichler <wogri@wogri.com> wrote:

> 
> 
> On 06/21/2013 01:46 PM, Remy Mudingay wrote:
>> All correct except that you need to classify from the prio class to the leaf qdisc, which in your case is tbf.
>> I strongly recommend testing this further.
> 
> I did test it now. It seems to be a discrepancy between my kernel and my
> userland tools.
> 
> I tested it on a debian stable stock kernel, now. and things behave as
> they should.
> 
>> 1. Create the prio qdisc with priomap but without attaching leaf qdisc's to the prio classes.
> 
> I see differentiation.
> 
>> 2. Then test again by attaching qdiscs to your prio class.
> 
> I see differentiation, also with qdiscs in my prio classes, and even
> without classification rules.
> 
>> 3. Finally test using your current setup but include classification rules.  
> 
> Don't matter anymore. I know that I've been fooled by tc :)
> 
>> Test 1 and 3 will both work.
>> 
>> Remy.
> 
> Even Test 2 worked.
> Thank you for your help.
> 
> Wolfgang
> 
>> 
>> Sent from my iPhone 
>> 
>> On Jun 21, 2013, at 11:09, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
>> 
>>> On Jun 21, 2013, at 11:03 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>>> 
>>>> Thanks for providing more info.
>>>> Firstly, the checksum error stems from tcp offloading on the local interface (wan0).
>>> 
>>> Thank you. I didn't know about TOE and it's implication before. 
>>> 
>>>> However, the prio qdisc does require the addition of filters when attaching qdisc's to prio classes. Take a look at the following  example :
>>>> 
>>>> http://d3s.mff.cuni.cz/~ceres/sch/osy/text/ch06s02s04.html
>>> 
>>> wow. Did I misunderstand the prio-qdisc? 
>>> from man tc-prio
>>> 
>>> CLASSIFICATION
>>>      Three methods are available to PRIO to determine in which band a packet will be enqueued.
>>> …
>>>      with the priomap
>>>             Based on the packet priority, which in turn is derived from the Type of Service assigned to the packet.
>>> 
>>>             The priomap maps the priority of a packet to a class. The priority can either be  set  directly  from  userspace,  or  be
>>>             derived from the Type of Service of the packet.
>>> 
>>>             Determines  how  packet priorities, as assigned by the kernel, map to bands. Mapping occurs based on the TOS octet of the
>>>             packet, which looks like this:
>>> … 
>>> 
>>> to me it seems to be the case that the prio-qdisc would be intelligent enouth to priorize for me. 
>>> 
>>> 
>>>> Remy
>>>> On Jun 21, 2013, at 7:42, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
>>>> 
>>>>> 
>>>>> -- 
>>>>> http://www.wogri.at
>>>>> 
>>>>> On Jun 21, 2013, at 05:11 , Remy Mudingay <remy.mudingay@gmail.com> wrote:
>>>>> 
>>>>>> Hi Wolfgang,
>>>>>> 
>>>>>> Can you post the output from 'tc -s class dev wan0' and also include
>>>>>> all filters for matching scp, ssh and telnet.
>>>>> 
>>>>> Hi Remy, 
>>>>> 
>>>>> tc -s class show dev wan0
>>>>> class prio 1:1 parent 1: leaf 10: 
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>>>> backlog 0b 0p requeues 0 
>>>>> class prio 1:2 parent 1: leaf 20: 
>>>>> Sent 97189 bytes 678 pkt (dropped 11, overlimits 409 requeues 0) 
>>>>> backlog 0b 0p requeues 0 
>>>>> class prio 1:3 parent 1: leaf 30: 
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>>>>> backlog 0b 0p requeues 0 
>>>>> class tbf 20:1 parent 20: 
>>>>> 
>>>>> (this is after doing some fiddling with an interactive ssh session). 
>>>>> There are no filters (neither tc nor iptables mangles), as far as I've understood prio should priorize according to it's priomap. 
>>>>> 
>>>>> here's a tcpdump: 
>>>>> 
>>>>> 05:39:28.048426 IP (tos 0x10, ttl 64, id 60303, offset 0, flags [DF], proto TCP (6), length 52)
>>>>>  x.x.x.x.50356 > y.y.y.y.22: Flags [.], cksum 0x545e (incorrect -> 0xc7ca), seq 0, ack 433, win 653, options [nop,nop,TS val 31327537 ecr 1639911627], length 0
>>>>> 
>>>>> oh. and maybe this is my issue: i see an incorrect checksum here on all those packets. maybe this is causing the troubles. is there a chance that this is happening because my interface is bridged? hm. I wonder how to deal with this (the actual communication works, btw). 
>>>>> 
>>>>> Wolfgang
>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Remy
> 
> 
> -- 
> http://www.wogri.com
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html