Fair queuing with htb

Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1: Type: Text/Plain, Size: 1503 bytes --]

Hello,

I try to make fair queuing with htb for the http protocol but my filters seem 
doesn't work.

The packets are marked by Squid (tcp_outgoing_mark, a feature of the latest 
version of squid enables this).

I have verified in the iptables logs that the packet are really marked. Here 
an example :

Dec 22 22:05:05 batman kernel: [1494086.818746] IN= OUT=eth0 SRC=172.17.0.8 
DST=172.17.0.10 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=65025 DF PROTO=TCP 
SPT=40407 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 MARK=0x1

It seems work inasmuch as we can see "MARK=0x1".

Below I paste my script. The relevant point is the filters : indeed, whatever 
the way I configure the class 1:20 and 1:10, it changes nothing.

Have I done a misconfiguration ?

I precise my server uses Debian Squeeze i386, and libnetfilter-conntrack3 is 
installed.

#!/bin/bash
INT="eth0"
BAND="12000kbit"
RATE="63kbit"

tc qdisc del dev $INT root
tc qdisc add dev $INT root handle 1: htb default 30

tc class add dev $INT parent 1: classid 1:1 htb rate 100mbit burst 15k 
tc class add dev $INT parent 1:1 classid 1:10 htb rate 63kbit ceil 1200kbit 
burst 15k 
tc class add dev $INT parent 1:1 classid 1:30 htb rate 88mbit burst 15k 
tc class add dev $INT parent 1:1 classid 1:20 htb rate $RATE ceil $RATE burst 
15k 

tc filter add dev $INT parent 1:0 protocol ip handle 0x1 fw flowid 1:10
tc filter add dev $INT protocol ip handle 0x55 fw flowid 1:20

Thanks for your help,
Grégoire Leroy


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Fair queuing with htb

[Julien Vehent]

On Thu, 23 Dec 2010 02:26:21 +0100, Grégoire Leroy <gregoire.leroy@hyperthese.net> wrote:
> Hello,
> 
> I try to make fair queuing with htb for the http protocol but my filters seem
> doesn't work.
> 
> The packets are marked by Squid (tcp_outgoing_mark, a feature of the latest 
> version of squid enables this).
> 
> I have verified in the iptables logs that the packet are really marked. Here 
> an example :
> 
> Dec 22 22:05:05 batman kernel: [1494086.818746] IN= OUT=eth0 SRC=172.17.0.8 
> DST=172.17.0.10 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=65025 DF PROTO=TCP 
> SPT=40407 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 MARK=0x1
> 
> It seems work inasmuch as we can see "MARK=0x1".
> 
> Below I paste my script. The relevant point is the filters : indeed, whatever
> the way I configure the class 1:20 and 1:10, it changes nothing.
> 
> Have I done a misconfiguration ?
> 
> I precise my server uses Debian Squeeze i386, and libnetfilter-conntrack3 is 
> installed.
> 
> #!/bin/bash
> INT="eth0"
> BAND="12000kbit"
> RATE="63kbit"
> 
> tc qdisc del dev $INT root
> tc qdisc add dev $INT root handle 1: htb default 30
> 
> tc class add dev $INT parent 1: classid 1:1 htb rate 100mbit burst 15k 
> tc class add dev $INT parent 1:1 classid 1:10 htb rate 63kbit ceil 1200kbit 
> burst 15k 
> tc class add dev $INT parent 1:1 classid 1:30 htb rate 88mbit burst 15k 
> tc class add dev $INT parent 1:1 classid 1:20 htb rate $RATE ceil $RATE burst
> 15k 
> 
> tc filter add dev $INT parent 1:0 protocol ip handle 0x1 fw flowid 1:10
> tc filter add dev $INT protocol ip handle 0x55 fw flowid 1:20
> 

Not sure about the handle 0x1. I usually only use the decimal value.

Try the following filters:

tc filter add dev $INT parent 1:0 protocol ip prio 1 handle 1 fw flowid 1:10
tc filter add dev $INT parent 1:0 protocol ip prio 2 handle 55 fw flowid 1:20


and check your filters and classes with the following commands:

# tc -s filter show dev eth0
# tc -s class show dev eth0

Julien

Re: Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1: Type: Text/Plain, Size: 2375 bytes --]

> Try the following filters:
> 
> tc filter add dev $INT parent 1:0 protocol ip prio 1 handle 1 fw flowid
> 1:10 tc filter add dev $INT parent 1:0 protocol ip prio 2 handle 55 fw
> flowid 1:20
> 
> 
> and check your filters and classes with the following commands:
> 
> # tc -s filter show dev eth0
> # tc -s class show dev eth0
> 

# tc -s filter show dev eth0
filter parent 1: protocol ip pref 1 fw 
filter parent 1: protocol ip pref 1 fw handle 0x1 classid 1:10 
filter parent 1: protocol ip pref 2 fw 
filter parent 1: protocol ip pref 2 fw handle 0x37 classid 1:20 

# tc -s class show dev eth0

class htb 1:1 root rate 100000Kbit ceil 100000Kbit burst 15337b cburst 1600b 
 Sent 908323001 bytes 680443 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 2976bit 2pps backlog 0b 0p requeues 0 
 lended: 38250 borrowed: 0 giants: 0
 tokens: 19047 ctokens: 1860

class htb 1:10 parent 1:1 prio 0 rate 63000bit ceil 1200Kbit burst 15Kb cburst 
1599b 
 Sent 2769846 bytes 41947 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 8bit 0pps backlog 0b 0p requeues 0 
 lended: 3697 borrowed: 38250 giants: 0
 tokens: -56194 ctokens: 144355

class htb 1:20 parent 1:1 prio 0 rate 63000bit ceil 63000bit burst 15Kb cburst 
1599b 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 30476187 ctokens: 3174593

class htb 1:30 parent 1:1 prio 0 rate 88000Kbit ceil 88000Kbit burst 15Kb 
cburst 1584b 
 Sent 905553155 bytes 638496 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 2968bit 2pps backlog 0b 0p requeues 0 
 lended: 80486 borrowed: 0 giants: 0
 tokens: 21656 ctokens: 2109


It seems there are packets which pass through the class 1:10.
However, I've dowloaded a 100Mo file, so it seems that most of packets pass 
through the default class...

The cause could be my design...

I have a pc which downloads a file on a web server, through the proxy server. 
They are all on the same network. The proxy server has only one network card 
(eth0).

They are linked by 100Mbit links. Squid mark packets from my pc and send them 
to netfilter.

Considering these informations, have I miss something in my tc configuration ? 
Must I use iptables to mark incoming traffic from the web server ?

Thanks for your help,
Grégoire Leroy

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Fair queuing with htb

[Andrew Beverley]

On Thu, 2010-12-23 at 13:30 +0100, Grégoire Leroy wrote:
> > Try the following filters:
> > 
> > tc filter add dev $INT parent 1:0 protocol ip prio 1 handle 1 fw flowid
> > 1:10 tc filter add dev $INT parent 1:0 protocol ip prio 2 handle 55 fw
> > flowid 1:20
> > 
> > 
> > and check your filters and classes with the following commands:
> > 
> > # tc -s filter show dev eth0
> > # tc -s class show dev eth0
> > 
> 
> # tc -s filter show dev eth0
> filter parent 1: protocol ip pref 1 fw 
> filter parent 1: protocol ip pref 1 fw handle 0x1 classid 1:10 
> filter parent 1: protocol ip pref 2 fw 
> filter parent 1: protocol ip pref 2 fw handle 0x37 classid 1:20 
> 
> # tc -s class show dev eth0
> 
> class htb 1:1 root rate 100000Kbit ceil 100000Kbit burst 15337b cburst 1600b 
>  Sent 908323001 bytes 680443 pkt (dropped 0, overlimits 0 requeues 0) 
>  rate 2976bit 2pps backlog 0b 0p requeues 0 
>  lended: 38250 borrowed: 0 giants: 0
>  tokens: 19047 ctokens: 1860
> 
> class htb 1:10 parent 1:1 prio 0 rate 63000bit ceil 1200Kbit burst 15Kb cburst 
> 1599b 
>  Sent 2769846 bytes 41947 pkt (dropped 0, overlimits 0 requeues 0) 
>  rate 8bit 0pps backlog 0b 0p requeues 0 
>  lended: 3697 borrowed: 38250 giants: 0
>  tokens: -56194 ctokens: 144355
> 
> class htb 1:20 parent 1:1 prio 0 rate 63000bit ceil 63000bit burst 15Kb cburst 
> 1599b 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  rate 0bit 0pps backlog 0b 0p requeues 0 
>  lended: 0 borrowed: 0 giants: 0
>  tokens: 30476187 ctokens: 3174593
> 
> class htb 1:30 parent 1:1 prio 0 rate 88000Kbit ceil 88000Kbit burst 15Kb 
> cburst 1584b 
>  Sent 905553155 bytes 638496 pkt (dropped 0, overlimits 0 requeues 0) 
>  rate 2968bit 2pps backlog 0b 0p requeues 0 
>  lended: 80486 borrowed: 0 giants: 0
>  tokens: 21656 ctokens: 2109
> 
> 
> It seems there are packets which pass through the class 1:10.
> However, I've dowloaded a 100Mo file, so it seems that most of packets pass 
> through the default class...
> 
> The cause could be my design...
> 
> I have a pc which downloads a file on a web server, through the proxy server. 
> They are all on the same network. The proxy server has only one network card 
> (eth0).
> 
> They are linked by 100Mbit links. Squid mark packets from my pc and send them 
> to netfilter.
> 
> Considering these informations, have I miss something in my tc configuration ? 
> Must I use iptables to mark incoming traffic from the web server ?

I suspect that your packets are not being marked correctly. If some of
the packets are going into the '10' class, but the majority are not,
then for some reason the majority are not being captured.

Please provide some more information on your setup (probably better to
the thread on the Squid list), or maybe do some closer inspection of
your packets MARK value using the LOG target.

Andy

Re: Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1.1: Type: Text/Plain, Size: 1118 bytes --]

> I suspect that your packets are not being marked correctly. If some of
> the packets are going into the '10' class, but the majority are not,
> then for some reason the majority are not being captured.
> 
> Please provide some more information on your setup (probably better to
> the thread on the Squid list), or maybe do some closer inspection of
> your packets MARK value using the LOG target.
> 
> Andy

I've attached a graph which explains what are the marked packets.

The general goal is to do a QoS based on user ip. If I had no proxy, it would 
be easy. However, since I've a proxy, my firewall sees the proxy ip, not the 
users IP.

So, I want to use Squid to mark packets according to the user ip, in order to 
apply my QoS rules in function of mark.

My logs say that marked packets are packets from squid to web server.

Another way to do what I want to do would be add an address by user on my 
network interface, use the tcp_outgoing_address directive on squid, and manage 
my QoS with the firewall on my gateway, based on the ip used by the proxy. But 
it would be... heavy.

[-- Attachment #1.2: proxy.odg --]
[-- Type: application/vnd.oasis.opendocument.graphics, Size: 11294 bytes --]

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Fair queuing with htb

[Antoine Souques]

> I've attached a graph which explains what are the marked packets.
>
Your design is wrong. You mark the upload traffic, when the main http 
traffic is the download traffic. That is why your QoS seems ineffective


> The general goal is to do a QoS based on user ip. If I had no proxy, 
> it would
> be easy. However, since I've a proxy, my firewall sees the proxy ip, 
> not the
> users IP.
>
Where is your firewall ? Between the proxy and the webserver, or the 
otherside ?

In the first case, you can only mark the upload traffic (it's to late 
for the download traffic). You should use the conntrack module to mark a 
connection, and so, you will be able to mark the download traffic

Moreover, I don't understand why you don't have access to your user 
addresses. You use mark, so your firewall and your proxy are running on 
the same box. So, when the download traffic leaves your proxy/firewall, 
the destination adsress is the user address. tc is called when a packet 
is send to the network, or when a packet arrive. So you can do IP based 
QoS.

Antoine

Re: Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1: Type: Text/Plain, Size: 1732 bytes --]

Le Saturday 25 December 2010 03:35:12, Antoine Souques a écrit :
> > I've attached a graph which explains what are the marked packets.
> 
> Your design is wrong. You mark the upload traffic, when the main http
> traffic is the download traffic. That is why your QoS seems ineffective

In this case I have probably misunderstood the goal of the tcp_outgoing_mark 
squid directive. Andrew, in what purpose is it developped ?

> 
> > The general goal is to do a QoS based on user ip. If I had no proxy, it
> > would be easy. However, since I've a proxy, my firewall sees the proxy
> > ip, not the users IP.
> 
> Where is your firewall ? Between the proxy and the webserver, or the
> otherside ?
> 

Yes it is

> In the first case, you can only mark the upload traffic (it's to late
> for the download traffic). You should use the conntrack module to mark a
> connection, and so, you will be able to mark the download traffic

I thank it was the goal of the tcp_outgoing_mark squid directive (authored by 
Andrew).

> Moreover, I don't understand why you don't have access to your user
> addresses. You use mark, so your firewall and your proxy are running on
> the same box. So, when the download traffic leaves your proxy/firewall,
> the destination adsress is the user address. tc is called when a packet
> is send to the network, or when a packet arrive. So you can do IP based
> QoS.

The problem is if I limit the traffic between the proxy and users, then he 
won't any difference between the data downloaded from internet and the data 
which was in cache in squid.

I want to limit the rate only for non-cached data, so it seems relevant to 
apply QoS between proxy server and internet.


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Fair queuing with htb

[Amos Jeffries]

On 25/12/10 16:51, Grégoire Leroy wrote:
> Le Saturday 25 December 2010 03:35:12, Antoine Souques a écrit :
>>> I've attached a graph which explains what are the marked packets.
>>
>> Your design is wrong. You mark the upload traffic, when the main http
>> traffic is the download traffic. That is why your QoS seems ineffective
>
> In this case I have probably misunderstood the goal of the tcp_outgoing_mark
> squid directive. Andrew, in what purpose is it developped ?
>
>>
>>> The general goal is to do a QoS based on user ip. If I had no proxy, it
>>> would be easy. However, since I've a proxy, my firewall sees the proxy
>>> ip, not the users IP.
>>
>> Where is your firewall ? Between the proxy and the webserver, or the
>> otherside ?
>>
>
> Yes it is
>
>> In the first case, you can only mark the upload traffic (it's to late
>> for the download traffic). You should use the conntrack module to mark a
>> connection, and so, you will be able to mark the download traffic
>
> I thank it was the goal of the tcp_outgoing_mark squid directive (authored by
> Andrew).
>
>> Moreover, I don't understand why you don't have access to your user
>> addresses. You use mark, so your firewall and your proxy are running on
>> the same box. So, when the download traffic leaves your proxy/firewall,
>> the destination adsress is the user address. tc is called when a packet
>> is send to the network, or when a packet arrive. So you can do IP based
>> QoS.
>
> The problem is if I limit the traffic between the proxy and users, then he
> won't any difference between the data downloaded from internet and the data
> which was in cache in squid.
>
> I want to limit the rate only for non-cached data, so it seems relevant to
> apply QoS between proxy server and internet.
>

There are three ways to do that:

1) Delay_pools in Squid capping the Server bandwidth speeds based on any 
client info desired.

2) QoS between the clients and Squid using qos_flows. They mark traffic 
destined to the clients separated into flows based on the data source 
type; cache, sibling peer, parent peer, direct origin.

  3) Between Squid and the origin servers you need to mark and limit on 
arrival into the box and Squid is not involved, or is set to pass-thru 
the markings. Squid will be limited along with the client.

AYJ

Re: Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1: Type: Text/Plain, Size: 941 bytes --]

> There are three ways to do that:
> 
> 1) Delay_pools in Squid capping the Server bandwidth speeds based on any
> client info desired.
> 

I noticed that Delay_pools permit us to balance rate according to user ip, 
with a class 3 Delay_pool. However, it seems to be a simple limitation. I 
don't want that the traffic is limited for one user, I want that each users 
have a guaranted rate, and if the rate is not used, then he is equally 
balanced between users who need the traffic. Exactly what cbq and htb do. It 
seems Delay_pools is not able to do this.

> 2) QoS between the clients and Squid using qos_flows. They mark traffic
> destined to the clients separated into flows based on the data source
> type; cache, sibling peer, parent peer, direct origin.

It seems it's exactly what I need. I have just tested it and it works. I will 
test it furthermore next week.

Thanks for your help.

Grégoire Leroy



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: Fair queuing with htb

[Andrew Beverley]

On Sat, 2010-12-25 at 04:51 +0100, Grégoire Leroy wrote:
> Le Saturday 25 December 2010 03:35:12, Antoine Souques a écrit :
> > > I've attached a graph which explains what are the marked packets.
> > 
> > Your design is wrong. You mark the upload traffic, when the main http
> > traffic is the download traffic. That is why your QoS seems ineffective
> 
> In this case I have probably misunderstood the goal of the tcp_outgoing_mark 
> squid directive. Andrew, in what purpose is it developped ?
> 

tcp_outgoing_mark will mark packets going from squid to the webserver.
If you want to mark packets going from squid to your client then you
need to use clientside_mark instead.

> > 
> > > The general goal is to do a QoS based on user ip. If I had no proxy, it
> > > would be easy. However, since I've a proxy, my firewall sees the proxy
> > > ip, not the users IP.
> > 
> > Where is your firewall ? Between the proxy and the webserver, or the
> > otherside ?
> > 
> 
> Yes it is
> 
> > In the first case, you can only mark the upload traffic (it's to late
> > for the download traffic). You should use the conntrack module to mark a
> > connection, and so, you will be able to mark the download traffic
> 
> I thank it was the goal of the tcp_outgoing_mark squid directive (authored by 
> Andrew).
> 

If you're going to mark packets on the other side of Squid, then you'll
probably need to move the interface that your HTB qdisc is attached to.

> > Moreover, I don't understand why you don't have access to your user
> > addresses. You use mark, so your firewall and your proxy are running on
> > the same box. So, when the download traffic leaves your proxy/firewall,
> > the destination adsress is the user address. tc is called when a packet
> > is send to the network, or when a packet arrive. So you can do IP based
> > QoS.
> 
> The problem is if I limit the traffic between the proxy and users, then he 
> won't any difference between the data downloaded from internet and the data 
> which was in cache in squid.
> 
> I want to limit the rate only for non-cached data, so it seems relevant to 
> apply QoS between proxy server and internet.
> 

In which case you want qos_flows, as Amos has already pointed out.

Andy

Re: Fair queuing with htb

[Grégoire Leroy]

[-- Attachment #1: Type: Text/Plain, Size: 209 bytes --]

I use this for a network with 150 users and it works well.

I've explained here how we can do :
http://retenodus.wordpress.com/2010/12/23/fair-queueing-with-squid/

Thanks for your help,
Grégoire Leroy

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 230 bytes --]