tc and priority

tc and priority

[Fabio Marcone]

Hi all,
I'm studying traffic shaping and I need some informations.
I studied HTB and I had no problem to limit bandwidth (with and without 
borrowing). Then I would undestand how assign different priorities to 
packets: I test qdisc PRIO and prio paramenter in CBQ (and HTB) but 
without success.

with CBQ, I create 2 two classes with different priority and I filter 
traffic to send different traffic (source port) to a particular queue. 
Then I run 2 connections having different priority and I see that a 
connection stalls and the other uses all bandwidth then it succeeds the 
opposite.

The same behaviour I see using PRIO qdisc.

Instead, I would get that che higher priority connection uses all 
bandwidth until its queue is full (so che lower priority  connection 
waits until the other finish).
What's wrong? Suggestions?

Thanks,
Fabio

Re: tc and priority

[Anatoly Muliarski]

Hi Fabio,

You should do something like this:

tc qdisc add dev eth0 root handle 1: prio bands 3
tc qdisc add dev eth0 parent 1:1 handle 2 sfq perturb 10
tc qdisc add dev eth0 parent 1:2 handle 3 sfq perturb 10
tc qdisc add dev eth0 parent 1:3 handle 4 sfq perturb 10
tc filter add dev eth0 protocol ip parent 1: prio 1 <your
criteria_high> flowid 1:1
tc filter add dev eth0 protocol ip parent 1: prio 2 <your
criteria_middle> flowid 1:2
tc filter add dev eth0 protocol ip parent 1: prio 3 <your
criteria_low> flowid 1:3

That works in my system.


2009/5/20, Fabio Marcone <fabio.marcone@duet.it>:
> Hi all,
> I'm studying traffic shaping and I need some informations.
> I studied HTB and I had no problem to limit bandwidth (with and without
> borrowing). Then I would undestand how assign different priorities to
> packets: I test qdisc PRIO and prio paramenter in CBQ (and HTB) but without
> success.
>
> with CBQ, I create 2 two classes with different priority and I filter
> traffic to send different traffic (source port) to a particular queue. Then
> I run 2 connections having different priority and I see that a connection
> stalls and the other uses all bandwidth then it succeeds the opposite.
>
> The same behaviour I see using PRIO qdisc.
>
> Instead, I would get that che higher priority connection uses all bandwidth
> until its queue is full (so che lower priority  connection waits until the
> other finish).
> What's wrong? Suggestions?
>
> Thanks,
> Fabio

-- 
Best regards
Anatoly Muliarski

Re: tc and priority

[Fabio Marcone]

Thanks for your reply.
I have just testing your script and I have some questions...

Anatoly Muliarski wrote:
> Hi Fabio,
>
> You should do something like this:
>
> tc qdisc add dev eth0 root handle 1: prio bands 3
> tc qdisc add dev eth0 parent 1:1 handle 2 sfq perturb 10
> tc qdisc add dev eth0 parent 1:2 handle 3 sfq perturb 10
> tc qdisc add dev eth0 parent 1:3 handle 4 sfq perturb 10
> tc filter add dev eth0 protocol ip parent 1: prio 1 <your
> criteria_high> flowid 1:1
> tc filter add dev eth0 protocol ip parent 1: prio 2 <your
> criteria_middle> flowid 1:2
> tc filter add dev eth0 protocol ip parent 1: prio 3 <your
> criteria_low> flowid 1:3
>   
why do you set a priority on filters? In my tests I always have been 
used packet marking instead of "u32 match" method without prio 
parameter. It is the same with priority?

> That works in my system.
>   
With your script I get the same result as mine: 2 parallel connections 
(with different priority), one uses all bandwitdh and the other stalls 
alternatively.

perhaps there is a timeout mechanims that forse sending queued packets 
although lower priority?


Thanks in advance,
Fabio

Re: tc and priority

[Покотиленко Костик]

В Чтв, 21/05/2009 в 10:13 +0200, Fabio Marcone пишет:
> Thanks for your reply.
> I have just testing your script and I have some questions...
> 
> Anatoly Muliarski wrote:
> > Hi Fabio,
> >
> > You should do something like this:
> >
> > tc qdisc add dev eth0 root handle 1: prio bands 3
> > tc qdisc add dev eth0 parent 1:1 handle 2 sfq perturb 10
> > tc qdisc add dev eth0 parent 1:2 handle 3 sfq perturb 10
> > tc qdisc add dev eth0 parent 1:3 handle 4 sfq perturb 10
> > tc filter add dev eth0 protocol ip parent 1: prio 1 <your
> > criteria_high> flowid 1:1
> > tc filter add dev eth0 protocol ip parent 1: prio 2 <your
> > criteria_middle> flowid 1:2
> > tc filter add dev eth0 protocol ip parent 1: prio 3 <your
> > criteria_low> flowid 1:3
> >   
> why do you set a priority on filters? In my tests I always have been 
> used packet marking instead of "u32 match" method without prio 
> parameter. It is the same with priority?
> 
> > That works in my system.
> >   
> With your script I get the same result as mine: 2 parallel connections 
> (with different priority), one uses all bandwitdh and the other stalls 
> alternatively.
> 
> perhaps there is a timeout mechanims that forse sending queued packets 
> although lower priority?

This is exactly how PRIO works. Higher priority classes get dequeued
first, so if there is something to dequeue forever then the lower
priority classes would never dequeue.

-- 
Покотиленко Костик <casper@meteor.dp.ua>

Re: tc and priority

[Fabio Marcone]

Hi!

>> With your script I get the same result as mine: 2 parallel connections 
>> (with different priority), one uses all bandwitdh and the other stalls 
>> alternatively.
>>
>> perhaps there is a timeout mechanims that forse sending queued packets 
>> although lower priority?
>>     
>
> This is exactly how PRIO works. Higher priority classes get dequeued
> first, so if there is something to dequeue forever then the lower
> priority classes would never dequeue.
>
>   
but it's not behaviour shown by test. And I don't no why. 
Ex.
connection A -> higher priority
connection B -> lower priority

results:
time      | connection A | connection B
1 sec           32 KB/s             0 KB/s
10 sec         18 KB/s           15 KB/s
20 sec           2 KB/s           31 KB/s
30 sec           0 KB/s           32 KB/s
40 sec         15 KB/s           16 KB/s
50 sec         25 KB/s             7 KB/s
60 sec         33 KB/s             0 KB/s
    ...                 ....                    ....
and so on

do you understand? I don't know why...

suggestions are welcome.

Fabio
 

Re: tc and priority

[Покотиленко Костик]

В Чтв, 21/05/2009 в 13:13 +0200, Fabio Marcone пишет:
> Hi!
> 
> >> With your script I get the same result as mine: 2 parallel connections 
> >> (with different priority), one uses all bandwitdh and the other stalls 
> >> alternatively.
> >>
> >> perhaps there is a timeout mechanims that forse sending queued packets 
> >> although lower priority?
> >>     
> >
> > This is exactly how PRIO works. Higher priority classes get dequeued
> > first, so if there is something to dequeue forever then the lower
> > priority classes would never dequeue.
> >
> >   
> but it's not behaviour shown by test. And I don't no why. 
> Ex.
> connection A -> higher priority
> connection B -> lower priority
> 
> results:
> time      | connection A | connection B
> 1 sec           32 KB/s             0 KB/s
> 10 sec         18 KB/s           15 KB/s
> 20 sec           2 KB/s           31 KB/s
> 30 sec           0 KB/s           32 KB/s
> 40 sec         15 KB/s           16 KB/s
> 50 sec         25 KB/s             7 KB/s
> 60 sec         33 KB/s             0 KB/s
>     ...                 ....                    ....
> and so on
> 
> do you understand? I don't know why...
> 
> suggestions are welcome.

If you mean why connB get traffic - then it's probably connA doesn't
*always* fill its bandwide which makes it possible for connB to send.

Try to fullfil connA with several mass downloading connections.

-- 
Покотиленко Костик <casper@meteor.dp.ua>

Re: tc and priority

[Fabio Marcone]

Hi!
> If you mean why connB get traffic - then it's probably connA doesn't
> *always* fill its bandwide which makes it possible for connB to send.
>
> Try to fullfil connA with several mass downloading connections
How I can fullfil connection A queue?
I run 4 connections A and 1 connection B, and this one continue to send 
packets... at about the same rate than the others.

Fabio

Re: tc and priority

[Anatoly Muliarski]

2009/5/21, Fabio Marcone <fabio.marcone@duet.it>:
> Hi,
> >
> > Different priorities are actual for filters with the same parent,
> > lower ones are looking first.
> >
> >
> ok
>
> > Check output:
> > tc -s filter show dev eth0
> > and
> > tc -s qdisc show dev eth0
> >
> >
> # tc -s -d qdisc show dev eth0
> qdisc prio 1: bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> Sent 7021292 bytes 4684 pkt (dropped 0, overlimits 0 requeues 0)
> rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc sfq 2: parent 1:1 limit 128p quantum 1514b flows 128/1024 perturb
> 10sec
> Sent 2997906 bytes 2003 pkt (dropped 0, overlimits 0 requeues 0)
> rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc sfq 3: parent 1:2 limit 128p quantum 1514b flows 128/1024 perturb
> 10sec
> Sent 720 bytes 8 pkt (dropped 0, overlimits 0 requeues 0)
> rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc sfq 4: parent 1:3 limit 128p quantum 1514b flows 128/1024 perturb
> 10sec
> Sent 4022666 bytes 2673 pkt (dropped 0, overlimits 0 requeues 0)
> rate 0bit 0pps backlog 0b 0p requeues 0
>
> # tc -s -d class show dev eth0
> class prio 1:1 parent 1: leaf 2:
> Sent 2997906 bytes 2003 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> class prio 1:2 parent 1: leaf 3:
> Sent 6082 bytes 49 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> class prio 1:3 parent 1: leaf 4:
> Sent 4022666 bytes 2673 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
>
> you see, class with higher priority sent less packets than class 1:3 (with
> lower priority). This is the proof that something is wrong, or I fall in
> error?
>
> class 1:2 receive packets from a ssh connection, used for test.
>
>
> Thanks,
> Fabio
>

I need additional info:
iptables -S
iptables -S -t mangle
and
tc -s filter show dev eth0.

There must be something wrong in your packet marking scheme.
What is your test environment?


-- 
Best regards
Anatoly Muliarski

Re: tc and priority

[Fabio Marcone]

Hi,
> I need additional info:
> iptables -S
> iptables -S -t mangle
> and
> tc -s filter show dev eth0.
>   
in this test I didn't use mangle table, but tc filter and u32 match:
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

# iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT

# tc -s filter show dev eth0
filter parent 1: protocol ip pref 1 u32
filter parent 1: protocol ip pref 1 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 
800 bkt 0 flowid 1:1  (rule hit 8503 success 4851)
  match 00500000/ffff0000 at 20 (success 4851 )
filter parent 1: protocol ip pref 3 u32
filter parent 1: protocol ip pref 3 u32 fh 801: ht divisor 1
filter parent 1: protocol ip pref 3 u32 fh 801::800 order 2048 key ht 
801 bkt 0 flowid 1:3  (rule hit 3652 success 3551)
  match 00510000/ffff0000 at 20 (success 3551 )

> There must be something wrong in your packet marking scheme.
> What is your test environment?
>   

client --- INTERNET--- router --- shaper1 --- server

where:
- client runs connections using wget on different ports
- shaper1 is a linux router that limit bandwith: 1500 Kbits (download 
for server and upload for client) and 300 kbit (upload for server and 
download for client)
- server: http server configured with your script

Fabio

>
>   

Re: tc and priority

[Anatoly Muliarski]

2009/5/21 Fabio Marcone <fabio.marcone@duet.it>:
>
>>
>> OK.
>> The starnge thing is:  match 00500000/ffff0000 at 20.
>> I suppose you try to match on dst_ip - there  must be 'at 16'!
>>
>
> what does it mean?
>
> this is my filter code:
> tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 80
> 0xffff flowid 1:1
> tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match ip sport 81
> 0xffff flowid 1:3
>
> and packets are queued in the right queue. If you prefer I can filter using
> iptables mangle, I think it's the same.
>
>
> Fabio

That's warmer :)
Try the next:
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip
protocol 6 0xFF ip sport 80 0xFFFF flowid 1:1
tc filter add dev eth0 protocol ip parent 1: prio 2 u32 match ip
protocol 6 0xFF ip sport 81 0xFFFF flowid 1:3
tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 flowid 1:2

-- 
Best regards
Anatoly Muliarski

Re: tc and priority

[Fabio Marcone]

Hi,
>> That's warmer :)
>> Try the next:
>> tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip
>> protocol 6 0xFF ip sport 80 0xFFFF flowid 1:1
>> tc filter add dev eth0 protocol ip parent 1: prio 2 u32 match ip
>> protocol 6 0xFF ip sport 81 0xFFFF flowid 1:3
>> tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 flowid 1:2
>>
>>     
I try to use use it but I get this error:
What is "ip"?
Usage: ...

What's wrong?

Very thanks,
Fabio

Re: tc and priority

[Anatoly Muliarski]

Hi Fabio,

2009/5/22 Fabio Marcone <fabio.marcone@duet.it>:
> Hi,
>>>
>>> That's warmer :)
>>> Try the next:
>>> tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip
>>> protocol 6 0xFF ip sport 80 0xFFFF flowid 1:1
>>> tc filter add dev eth0 protocol ip parent 1: prio 2 u32 match ip
>>> protocol 6 0xFF ip sport 81 0xFFFF flowid 1:3
>>> tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0
>>> flowid 1:2
>>>
>>>
>
> I try to use use it but I get this error:
> What is "ip"?
> Usage: ...
>
> What's wrong?
>
> Very thanks,
> Fabio

Sorry, there was my error.
The correct version:
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 classid 1:1
match ip protocol 6 0xFF match ip sport 80 0xFFFF
tc filter add dev eth0 protocol ip parent 1: prio 2 u32 classid 1:3
match ip protocol 6 0xFF ip sport 81 0xFFFF
tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 flowid 1:2


-- 
Best regards
Anatoly Muliarski

Re: tc and priority

[Fabio Marcone]

Hi,
>
> Sorry, there was my error.
> The correct version:
> tc filter add dev eth0 protocol ip parent 1: prio 1 u32 classid 1:1
> match ip protocol 6 0xFF match ip sport 80 0xFFFF
> tc filter add dev eth0 protocol ip parent 1: prio 2 u32 classid 1:3
> match ip protocol 6 0xFF ip sport 81 0xFFFF
> tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 flowid 1:2
Very thanks, I will try it soon.

Fabio

Re: tc and priority

[Fabio Marcone]

Hi,
I test this script:
#!/bin/bash

tc qdisc del dev eth0 root
iptables -P INPUT ACCEPT

tc qdisc add dev eth0 root handle 1: prio bands 3
tc qdisc add dev eth0 parent 1:1 handle 2 sfq perturb 10
tc qdisc add dev eth0 parent 1:2 handle 3 sfq perturb 10
tc qdisc add dev eth0 parent 1:3 handle 4 sfq perturb 10
#tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 
80 0xffff flowid 1:1
##tc filter add dev eth0 protocol ip parent 1: prio 2 <your 
criteria_middle> flowid 1:2
#tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match ip sport 
81 0xffff flowid 1:3
tc filter add dev eth0 protocol ip parent 1: prio 1 u32 classid 1:1 
match ip protocol 6 0xFF match ip sport 80 0xFFFF
tc filter add dev eth0 protocol ip parent 1: prio 2 u32 classid 1:3 
match ip protocol 6 0xFF match ip sport 81 0xFFFF
tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 
flowid 1:2


I run 4 high priority processes (classid 1:1) and 1 low priority one 
(classid 1:3).  These are the result:
high proc1: 19.5K/s
high proc2: 5.71K/s
high proc3: 738B/s
high proc4: 12.8K/s

low proc: 5.8 K/s

I don't undestand: low process didn't stall and a high process used a 
datarate << than low process.


Help, help, help!

Fabio

Re: tc and priority

[Anatoly Muliarski]

Hi,

Check out:
 tc -s filter show dev eth0.

Where do you using prio qdisc in your testing scheme
( client --- INTERNET--- router --- shaper1 --- server )?

As a hint - prio qdisc sends packets at WIRE speed, if you need to
shape speed you should use tbf or htb/hfsc qdiscs in leaf classes(
1:1,1:2,1:3 )


2009/5/25, Fabio Marcone <fabio.marcone@duet.it>:
> Hi,
> I test this script:
> #!/bin/bash
>
> tc qdisc del dev eth0 root
> iptables -P INPUT ACCEPT
>
> tc qdisc add dev eth0 root handle 1: prio bands 3
> tc qdisc add dev eth0 parent 1:1 handle 2 sfq perturb 10
> tc qdisc add dev eth0 parent 1:2 handle 3 sfq perturb 10
> tc qdisc add dev eth0 parent 1:3 handle 4 sfq perturb 10
> #tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip sport 80
> 0xffff flowid 1:1
> ##tc filter add dev eth0 protocol ip parent 1: prio 2 <your criteria_middle>
> flowid 1:2
> #tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match ip sport 81
> 0xffff flowid 1:3
> tc filter add dev eth0 protocol ip parent 1: prio 1 u32 classid 1:1 match ip
> protocol 6 0xFF match ip sport 80 0xFFFF
> tc filter add dev eth0 protocol ip parent 1: prio 2 u32 classid 1:3 match ip
> protocol 6 0xFF match ip sport 81 0xFFFF
> tc filter add dev eth0 protocol ip parent 1: prio 3 u32 match u8 0 0 flowid
> 1:2
>
>
> I run 4 high priority processes (classid 1:1) and 1 low priority one
> (classid 1:3).  These are the result:
> high proc1: 19.5K/s
> high proc2: 5.71K/s
> high proc3: 738B/s
> high proc4: 12.8K/s
>
> low proc: 5.8 K/s
>
> I don't undestand: low process didn't stall and a high process used a
> datarate << than low process.
>
>
> Help, help, help!
>
>
> Fabio


-- 
Best regards
Anatoly Muliarski

Re: tc and priority

[Fabio Marcone]

Hi,
I run the same test witth this configuration:
 client ---------- server

with a 100 Mbs link. In this case high priority process has about 11 
MB/s and  low priority one has 400 KB/s. EUREKA! here the prio is correct!

I think that in the other tests using:
> ( client --- INTERNET--- router --- shaper1 --- server )
>   
with prio set in the server, high priority queue is not always full, maybe.

Now I will run some other test to deep understand the situation.

Thanks a lot,
Fabio