[half_OT]Traffic shaping with tc and iptables

[half_OT]Traffic shaping with tc and iptables

[Marco Coda]

Hi list!

I'm optimizing a mail server (postfix+cyrus) and I want to limit the
maximum outbound bandwith of the port 25 only, for not saturate all
the bandwith of the office.
For example, here we have 2 Mb in upload. I want to leave to the server 1 Mb max

Reading the LARTC and other articles on the web I tried some filtering
with tc command and they actually work, but they do no respect my
policies: if I write 10kbit the maximum speed is 48x (costant) fast!
480kbps instaead of 10kbps

Here there is the script I ran:
############################################################################
# path of iptables
IPTABLES=/sbin/iptables
# ethernet interface that I want to limit
nic=eth0
# Port I want to limit
port=25
# Maximum upload limit...
upload=10
# ...and relative unit
unit=kbit
# Maximum LAN speed (we have 100Mb switch so..)
maxspeed=100Mbit
# weight of the limit...
weight=1
# ...and relative unit
unit2=kbit

tc qdisc del dev $nic root && iptables -t mangle -F
$IPTABLES -t mangle -A OUTPUT -p tcp --sport $port -j MARK --set-mark 1
tc qdisc add dev $nic root handle 10: cbq bandwidth $maxspeed avpkt 1000 mpu 64
tc class add dev $nic parent 10:0 classid 10:1 cbq rate $upload$unit
weight $weight$unit2 allot 1514 prio 1 avpkt 1000 bounded
tc filter add dev $nic parent 10:0 protocol ip handle 1 fw flowid 10:1
######################################################################################

I tried also with the u32 filter, instead of iptables + handle fw, but
with the same speed moltiplicative constant:

################################################################################
tc qdisc del dev $nic root && iptables -t mangle -F
$IPTABLES -t mangle -A OUTPUT -p tcp --sport $port -j MARK --set-mark 1
tc qdisc add dev $nic root handle 10: cbq bandwidth $maxspeedt avpkt 1000 mpu 64
tc filter add dev $nic protocol ip parent 10: prio 1 u32 match ip
sport $port 0xffff flowid 10:1
#################################################################################

For calculating the real speed I use the "size" value (in bytes) that
I find in postfix's log, then I convert in bits and I divide for the
number of seconds between the connection and the disconnection of the
mail client.

Any ideas for resolving the problem?

Re: [half_OT]Traffic shaping with tc and iptables

[Gáspár Lajos]

Hi!

> For example, here we have 2 Mb in upload. I want to leave to the server 1 Mb max
>
>

> Here there is the script I ran:
> ############################################################################
> # path of iptables
> IPTABLES=/sbin/iptables
> # ethernet interface that I want to limit
> nic=eth0
> # Port I want to limit
> port=25
> # Maximum upload limit...
> upload=10
> # ...and relative unit
> unit=kbit
Or: upload='1Mbit'
> # Maximum LAN speed (we have 100Mb switch so..)
> maxspeed=100Mbit
I would set: bandwidth='2Mbit'
> # weight of the limit...
> weight=1
Not really needed...
> # ...and relative unit
> unit2=kbit
>
> tc qdisc del dev $nic root&&  iptables -t mangle -F
> $IPTABLES -t mangle -A OUTPUT -p tcp --sport $port -j MARK --set-mark 1
$IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --sport $port -j MARK 
--set-mark 1
> tc qdisc add dev $nic root handle 10: cbq bandwidth $maxspeed avpkt 1000 mpu 64
tc qdisc add dev $nic root handle 1:0 cbq bandwidth $bandwidth avpkt 
1000 mpu 64
> tc class add dev $nic parent 10:0 classid 10:1 cbq rate $upload$unit
> weight $weight$unit2 allot 1514 prio 1 avpkt 1000 bounded
tc class add dev $nic parent 1:0 classid 1:1 cbq rate $upload allot 1514 
prio 1 avpkt 1000 bounded isolated
# Check if you need isolated.... man tc-cbq

> tc filter add dev $nic parent 10:0 protocol ip handle 1 fw flowid 10:1
tc filter add dev $nic parent 1:0 ip handle fw flowid 1:1


Swifty

Re: [half_OT]Traffic shaping with tc and iptables

[Gáspár Lajos]

Sorry... :D

tc filter add dev $nic parent 1:0 protocol ip handle fw flowid 1:1

Swifty

Re: [half_OT]Traffic shaping with tc and iptables

[Marco Coda]

Ok, now i ran, as you adviced me:

2011/9/8 Gáspár Lajos <swifty@freemail.hu>:

> IPTABLES=/sbin/iptables
> nic=eth0
> port=25
> upload=10kbit
> bandwith=2Mbit
>
> $IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --sport $port -j MARK --set-mark 1
> tc qdisc add dev $nic root handle 1:0 cbq bandwidth $bandwidth avpkt 1000 mpu 64
> tc class add dev $nic parent 1:0 classid 1:1 cbq rate $upload allot 1514 prio 1 avpkt 1000 bounded isolated
>tc filter add dev $nic parent 1:0 protocol ip handle fw flowid 1:1

But with this script I obtain the same result:
I tried to send a mail with an attachment of 3 MB. Between the "client
connection" and the "removed from queue", passed 1'12''
With tail /var/log/maillog | grep "size=" I saw that the real mail
size is 4317831B, so the real speed is  (4317831*8)[bits] / 72[s] =
479759, 468 Kbps, not 10 Kbps as I set..
I cannot understand the reason of this differences...

Re: [half_OT]Traffic shaping with tc and iptables

[Gáspár Lajos]

2011-09-08 12:54 keltezéssel, Marco Coda írta:
>> IPTABLES=/sbin/iptables
>> nic=eth0
>> port=25
>> upload=10kbit
>> bandwith=2Mbit
>>
>> $IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --sport $port -j MARK --set-mark 1
AAAAAAAAAAArrrrrghhhh!!!! :D

If you SEND a mail FROM your server to an ANOTHER server then the 
DESTINATION port will be 25 !!!! :D

$IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --dport $port -j MARK 
--set-mark 1

Try again!!! :D

Swifty

Re: [half_OT]Traffic shaping with tc and iptables

[Marco Coda]

2011/9/8 Gáspár Lajos <swifty@freemail.hu>:

> $IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --dport $port -j MARK --set-mark 1

With this rule it does not work at all.. :(
At least before, even if I understand that concectually is not
correct, with --sport, traffic was shaped, in some manner, whth a
strange rate..

Re: [half_OT]Traffic shaping with tc and iptables

[Gáspár Lajos]

2011-09-08 14:10 keltezéssel, Marco Coda írta:
> With this rule it does not work at all.. :(
What do you mean it does not work at all???
Does the packet counter shows that the rule got any hit??? (iptables 
-vnL OUTPUT -t mangle)
Can you see the mark in the conntrack??? (conntrack -L | grep 'mark=1')
> At least before, even if I understand that concectually is not
> correct, with --sport, traffic was shaped, in some manner, whth a
> strange rate..
- You can only shape those traffic that leaves your system...
- If you send an e-mail then you connect from your system (from a random 
port) to a mail server (to 25)...
- If you set your upload limit to 10kbit then you can send 1,25KByte per 
sec. (It is veeerrryy slooow.) Would you try with my proposed settings???

Swifty

Re: [half_OT]Traffic shaping with tc and iptables

[Nikolay Kichukov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
tc does not require iptables to shape traffic at all. So why bothering?

Cheers,
- -Nik

On 09/08/2011 03:10 PM, Marco Coda wrote:
> 2011/9/8 Gáspár Lajos <swifty@freemail.hu>:
> 
>> $IPTABLES -t mangle -A OUTPUT -p tcp -o $nic --dport $port -j MARK --set-mark 1
> 
> With this rule it does not work at all.. :(
> At least before, even if I understand that concectually is not
> correct, with --sport, traffic was shaped, in some manner, whth a
> strange rate..
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOaLfXAAoJEDFLYVOGGjgXG7UIAKW+y7j/t5R25HHYRYamaX5e
xZ5L+2TDS4KSsAjGQXq6ZvRhm/Z1QUWSL4U0qBbPZQNuw1KP2d/z+u6yuFe1XBPK
DBedlFlg6l7+O8H0ruyv8RXULRDTOkzQY9jXkh4qV/eJQy36+4Yg4D4JIwgqlf2r
MszARiQycw4Za6zX630tOwnl8bbZ30ifJUs6ZabZVrV45tY0JXV2y7lRuajBLb8q
FoUPJPA7hxYwR6VzzNQJN0/WWZ4VDXKV0nGPUitCtdZvsO1iUb74ZeR+bpxTQZr6
4Y52UaUp4XUEuHr69BPt0C+C35Pk6O9hSUaD4UUWQU1zB7quC2wlrZ8ZqtJd/c8=
=NrKl
-----END PGP SIGNATURE-----

Re: [half_OT]Traffic shaping with tc and iptables

[Marco Coda]

2011/9/8 Gáspár Lajos <swifty@freemail.hu>:


> - If you send an e-mail then you connect from your system (from a random
> port) to a mail server (to 25)...
> Would you try with my proposed settings???

I just tried it, with rare 1Mbit, bandwidth 2 Mbit and iptables with
--dport 25 and, even if the iptables rule is matched (I can see the
packet count measuring the right size of the mail), tc seems to ignore
those packets. I know that my postfix open a connection to another mta
from a pseudo-random port to 25, but with --dport option tc does not
consider these packets. Instead, with --sport option, I don't know
why, something is filtered...

> - If you set your upload limit to 10kbit then you can send 1,25KByte per
> sec. (It is veeerrryy slooow.)

In this moment I set this speed so I can test the server with small
attachments... When the script will be definitively complete,I'll set
the real values..


2011/9/8 Nikolay Kichukov <hijacker@oldum.net>:

> tc does not require iptables to shape traffic at all. So why bothering?

I want to limit only one port, not the entire interface. I don't want
other ports (such as pop3 or imap) limited
So I used iptables for marking and then tc filter handle fw for filtering..
How should i do?

Re: [half_OT]Traffic shaping with tc and iptables

[Gáspár Lajos]

2011-09-08 16:00 keltezéssel, Marco Coda írta:
> I just tried it, with rare 1Mbit, bandwidth 2 Mbit and iptables with
> --dport 25 and, even if the iptables rule is matched (I can see the
> packet count measuring the right size of the mail), tc seems to ignore
iptales rule matched -> that is good !!! :D
> those packets. I know that my postfix open a connection to another mta
> from a pseudo-random port to 25, but with --dport option tc does not
> consider these packets. Instead, with --sport option, I don't know
> why, something is filtered...
>
Do you see the connection in conntrack??? With the mark=1 value???
Maybe you are trying to set the whole tc on the wrong interface ?!? (As 
I mentioned before: You can shape the leaving traffic... On the 
interface that is used for the connection...)
>> - If you set your upload limit to 10kbit then you can send 1,25KByte per
>> sec. (It is veeerrryy slooow.)
> In this moment I set this speed so I can test the server with small
> attachments... When the script will be definitively complete,I'll set
> the real values..
Maybe too low values would disable the tc?!? (I am really not sure about 
it... :D )
>
> 2011/9/8 Nikolay Kichukov<hijacker@oldum.net>:
>
>> tc does not require iptables to shape traffic at all. So why bothering?
Yes, it can be done without iptables... But filtering in iptables 
"maybe" easier than in tc...
Currently it does not work at all... :(

Swifty

Re: [half_OT]Traffic shaping with tc and iptables

[Nikolay Kichukov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
#create engress htb shaper:
tc qdisc add dev $IF root handle 1: htb
#setup some iface limits
tc class add dev $IF parent 1: classid 1:1 htb rate ${MAXOUT}kbit ceil ${MAXOUT}kbit burst 50kb

#add subclass for your application attached to the root class 1:1 - 1:10, set proper ceil value!
tc class add dev $IF parent 1:1 classid 1:10 htb rate 128kbit ceil ${MAXOUT}kbit prio 1 burst 5kb

#then just add filters, in that case port 53 - DNS
tc filter add dev $IF parent 1:0 protocol ip prio 10 u32 match ip dport 53 0xffff flowid 1:10


you are good to go!
tc qdisc -s show dev $IF

will tell you what is going on.

HTH,
- -N

On 09/08/2011 05:00 PM, Marco Coda wrote:
> 2011/9/8 Gáspár Lajos <swifty@freemail.hu>:
> 
> 
>> - If you send an e-mail then you connect from your system (from a random
>> port) to a mail server (to 25)...
>> Would you try with my proposed settings???
> 
> I just tried it, with rare 1Mbit, bandwidth 2 Mbit and iptables with
> --dport 25 and, even if the iptables rule is matched (I can see the
> packet count measuring the right size of the mail), tc seems to ignore
> those packets. I know that my postfix open a connection to another mta
> from a pseudo-random port to 25, but with --dport option tc does not
> consider these packets. Instead, with --sport option, I don't know
> why, something is filtered...
> 
>> - If you set your upload limit to 10kbit then you can send 1,25KByte per
>> sec. (It is veeerrryy slooow.)
> 
> In this moment I set this speed so I can test the server with small
> attachments... When the script will be definitively complete,I'll set
> the real values..
> 
> 
> 2011/9/8 Nikolay Kichukov <hijacker@oldum.net>:
> 
>> tc does not require iptables to shape traffic at all. So why bothering?
> 
> I want to limit only one port, not the entire interface. I don't want
> other ports (such as pop3 or imap) limited
> So I used iptables for marking and then tc filter handle fw for filtering..
> How should i do?
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOaNfCAAoJEDFLYVOGGjgXIcsIAKTB5Azc5860kSvNmyLjaDTH
WgZfmiPzoyuSK88WhXaIVBXcwLgpBVVqkZZRV3AyXKQ/ucTGax6daDZdmINw+i53
YIkKzQCknaEff/WdVCfVi404OERxz/tzUwHAqN4/DsS7/h55XPkpmBEgUahIYeWP
3RQZ9mNFkzpdYWnoLefFgtgBjxecShocQ2wyRAybl4KJQnl+5tv+tTQqiOQ0t6Cz
aPyX4w26qaluQiSTQ6SXeJ846HWASjvAt3KIXaS1xc9c000OeGT0vHCLBf+I5whE
sghiHVMBqcF8IVs+s+2vVn200d2MSzfhtz2llYAiEqxExXOhQ1y6nm8k1XSqfu8=
=ip57
-----END PGP SIGNATURE-----

Re: [half_OT]Traffic shaping with tc and iptables

[Marco Coda]

2011/9/8 Gáspár Lajos <swifty@freemail.hu>:

> Do you see the connection in conntrack??? With the mark=1 value???
Unfortunately this server now is in a private and close lan, not
accessible ftom or to the Internet, so I cannot install conntrack
I set up a system with one dns, one client and another postfix for test

> Maybe too low values would disable the tc?!? (I am really not sure about
> it... :D )

Before I tried also wit different values: all the ten's multiples
untill 100, to see if the 48x factor is costant also with other speed.
And no, is not constant: it increase as an exponential curve!


2011/9/8 Nikolay Kichukov <hijacker@oldum.net>:

> #create engress htb shaper:
> tc qdisc add dev $IF root handle 1: htb
> #setup some iface limits
> tc class add dev $IF parent 1: classid 1:1 htb rate ${MAXOUT}kbit ceil ${MAXOUT}kbit burst 50kb
>
> #add subclass for your application attached to the root class 1:1 - 1:10, set proper ceil value!
> tc class add dev $IF parent 1:1 classid 1:10 htb rate 128kbit ceil ${MAXOUT}kbit prio 1 burst 5kb
>
> #then just add filters, in that case port 53 - DNS
> tc filter add dev $IF parent 1:0 protocol ip prio 10 u32 match ip dport 53 0xffff flowid 1:10

Your script seems work better than mine: the speed now is 20Kbps, and
not value as 480kbps as before with my commands!, even if I set
10kbits, both as ceil and rate:

IF=eth0
MAXOUT=10kbit

tc qdisc add dev $IF root handle 1: htb
tc class add dev $IF parent 1: classid 1:1 htb rate $MAXOUT ceil $MAXOUT
tc class add dev $IF parent 1:1 classid 1:10 htb rate 10kbit ceil $MAXOUT
tc filter add dev $IF parent 1:0 protocol ip prio 10 u32 match ip
dport 25 0xffff flowid 1:10

I erase priority and brust part, At the moment don't mind brusting the
connection or assign a priority.