tc and CONNMARK

tc and CONNMARK

[Fabio Marcone]

Hi!
I'm building a firewall/traffic shaper using tc and connmark but I note 
a strange behaviour.

Configuration:
eth0 - lan interface - rate 600kbit/s connection with mark X (download)
eth1 - wan interface - rate 600kbit/s connection with mark X (upload)

Starting a single connection (upload or download) traffic is limited 
correctly but if I start two connection (one in upload and the other in 
download) I note that the one in upload runs correctly (how in the 
previous test) but the one in download stalls and then send some packets 
and then stalls alternatively.

I don't understand why!

always (and only) the download connection has this behaviour.


two connection have the same mark (in netfilter table), is it a problem 
for the shaper?
also I think about ack packets but they have a very short size (66 
bytes) so I believe they are not the cause for this problem.

thanks in advance,
Fabio Marcone

Re: tc and CONNMARK

[Fabio Marcone]

Thanks for your reply,

> A first idea, without saw your rules, it's that you limit the wrong side
> of communication, however no one can help you without understand better
> your environment.
it's not possible because if I test only download flow or upload flow, 
limit rate is ok.

> Post your tc/iptables rules and commands.

I'm developing a firewall/router with some services and configuration 
it's not so easy to understand.

>
> P.s. Use two different mark for different update/download traffic
I'm using connmark so both flow of connection are marked with same mark 
(CONNMARK) but shaper rule are set on different interfaces:
mark 10 on eth0 -> classid 1:6
mark 10 on eth1 -> classid 2:6

where classes are build with these commands:
tc class add dev eth0 parent 1:1 classid 1:6 htb rate 600kbit ceil 
600kbit burst 6k cburst 64k quantum 1600
tc class add dev eth0 parent 2:1 classid 2:6 htb rate 600kbit ceil 
600kbit burst 6k cburst 64k quantum 1600

and filters are:
tc filter add dev eth0 parent 1: protocol ip handle 10 fw flowid 1:6
tc filter add dev eth1 parent 2: protocol ip handle 10 fw flowid 2:6

do you think is a connmark problem?

Re: tc and CONNMARK

[Michele Petrazzo - Unipex]

Fabio Marcone ha scritto:
> Thanks for your reply,
> 
>> A first idea, without saw your rules, it's that you limit the wrong
>>  side of communication, however no one can help you without 
>> understand better your environment.
> it's not possible because if I test only download flow or upload 
> flow, limit rate is ok.
> 
>> Post your tc/iptables rules and commands.
> 
> I'm developing a firewall/router with some services and configuration
>  it's not so easy to understand.
> 

Ok, but at least an environment where we can test or understand it's need!

>> 
>> P.s. Use two different mark for different update/download traffic
> I'm using connmark so both flow of connection are marked with same 
> mark (CONNMARK) but shaper rule are set on different interfaces: mark
>  10 on eth0 -> classid 1:6 mark 10 on eth1 -> classid 2:6

For me it's better if you use different marks.

After, where do you mark the packets (in iptables)? iptables/kernel
version?

> 
> where classes are build with these commands: tc class add dev eth0 
> parent 1:1 classid 1:6 htb rate 600kbit ceil 600kbit burst 6k cburst 
> 64k quantum 1600 tc class add dev eth0 parent 2:1 classid 2:6 htb 
> rate 600kbit ceil 600kbit burst 6k cburst 64k quantum 1600
> 
> and filters are: tc filter add dev eth0 parent 1: protocol ip handle 
> 10 fw flowid 1:6 tc filter add dev eth1 parent 2: protocol ip handle 
> 10 fw flowid 2:6
> 
> do you think is a connmark problem?

Normally I don't use connmark because when I try some time ago to use
it, I found some "not marked" problems, so I switch to classid. Better
and cleaner for me.

Michele

Re: tc and CONNMARK

[Fabio Marcone]

hello,
> For me it's better if you use different marks.

I note that packets are marked correctly but in the wan interface tc 
filter send them on the wrong class. is the same problem you had it?
>
> After, where do you mark the packets (in iptables)? iptables/kernel
> version?
I mark packets in iptables (mangle table PREROUTING chain):

iptables -t mangle -A PREROUTING -p TCP -m mac --mac-source 
xx:xx:xx:xx:xx:xx--dport 443 -j MARK --set-mark 8

iptables version: 1.4.3.2
kernel version: 2.6.29.3

both patched to use IMQ devices.
>
> Normally I don't use connmark because when I try some time ago to use
> it, I found some "not marked" problems, so I switch to classid. Better
> and cleaner for me.
what kind of problems? What do you mean with "I switch to classid" ?

thanks in advance,
Fabio

Re: tc and CONNMARK

[Richard Horton]

Just realised I'd sent this direct not via list... must remember to
check address lines before sending in future.

2009/10/6 Richard Horton <arimus.uk@googlemail.com>:
> 2009/10/6 Fabio Marcone <fabio.marcone@duet.it>:
>
>> what kind of problems? What do you mean with "I switch to classid" ?
>>
>
> Classid in this context I would assume to mean using the diff serv
> code point field (was the ToS field) in the IP Header and making
> queuing decisions based on the diffserv value rather than using
> marking in the packet structure.
>
> If you use Classid there are two routes you can go do;
>
> Route 1 uses 'class selectors' and provides 8 levels of discrimination.
> Route 2 goes down the full diffserv path and provides 3 levels of
> traffic - Expedited forwards, Assured Forwarding and 'Best' Effort
> (which is anything but best).
>
> The EF class is designed for real-time jitter (variation of delay)
> sensitive traffic, AF is designed for loss-intolerant traffic and
> provides 4 levels of forwarding each with 3 levels of drop precedence
> and BE is well... whatever is left over.
>
> Using the AFxy (x class, y drop precedence) you can define some
> complex schemes with each class having a guaranteed bandwidth with
> packets being dropped on the basis of their drop precedence...
>
>
> For instance to mark rtp and give it decent bandwidth/priority and sip
> slightly lower...
>
> (Using port 20000/20001 for rtp, 5060 for sip)
> iptables -A INPUT -t mangle -p udp -dport 5060 -j DSCP --set-dscp-class AF41
> iptables -A INPUT -t mangle -p udp -dport 20000 -j DSCP --set-dscp-class EF
> iptables -A INPUT -t mangle -p udp -dport 20001 -j DSCP --set-dscp class EF
>
> tc qdisc add dev eth0 root handle 1: htb default 10
> tc class add dev eth0 classid 1:10 parent 1: htb rate 10mbps ceil 10mbps prio 10
> tc class add dev eth0 classid 1:20 parent 1: htb rate 30mbps ceil 40mbps prio 5
> tc class add dev eth0 classid 1:30 parent 1: htb rate 60mbps ceil 100mbps prio 0
>
> tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 \
>   match ip tos 0x88 0xff flowid 1:20
> tc filter add dev eth0 parent 1:0 protocol ip prio 10 u32 \
>   match ip tos 0xB8 0xff flowid 1:30
>
>
>
> --
> Richard Horton
> Users are like a virus: Each causing a thousand tiny crises until the
> host finally dies.
> http://www.solstans.co.uk - Solstans Japanese Bobtails and Norwegian Forest Cats
> http://www.pbase.com/arimus - My online photogallery
>



-- 
Richard Horton
Users are like a virus: Each causing a thousand tiny crises until the
host finally dies.
http://www.solstans.co.uk - Solstans Japanese Bobtails and Norwegian Forest Cats
http://www.pbase.com/arimus - My online photogallery

Re: tc and CONNMARK

[Michele Petrazzo - Unipex]

Fabio Marcone ha scritto:
> hello,
>> For me it's better if you use different marks.
> 
> I note that packets are marked correctly but in the wan interface tc 
> filter send them on the wrong class. is the same problem you had it?
> 
> 
If I remember correctly, yes.

> iptables -t mangle -A PREROUTING -p TCP -m mac --mac-source 
> xx:xx:xx:xx:xx:xx--dport 443 -j MARK --set-mark 8
> 

Why not forward or postrouting? Do you choose prerouting for use the mac 
addrs?

> iptables version: 1.4.3.2 kernel version: 2.6.29.3
> 
> both patched to use IMQ devices.

Haven't you said this!
In _all_ my installations, I always skip to use imq. I don't know why,
but it's simple to use that are already included inside the vanilla kernel

>> 
>> Normally I don't use connmark because when I try some time ago to 
>> use it, I found some "not marked" problems, so I switch to classid.
>>  Better and cleaner for me.
> what kind of problems?

ip filter don't match my data. But it's true that I didn't lost a lot of
time following that solution because... I find CLASSIFY

> What do you mean with "I switch to classid" ?
> 

It's a wrong definition, sorry. The right one it CLASSIFY!

Simple example, assuming that you have 192.168.1.0/24, need to limit at 5mb
the ip .100 on both sides on flow and all the others goes at 1mbit all
together. eth0 lan, eth1 are wan.

tc qdisc add dev eth0 root handle 1: htb default 1000
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit ceil 100mbit
tc class add dev eth0 parent 1:1 classid 1:100 htb rate 1mbit ceil 5mbit
tc class add dev eth0 parent 1:1 classid 1:1000 htb rate 1mbit ceil 1mbit

tc qdisc add dev eth1 root handle 2: htb default 1000
tc class add dev eth1 parent 2: classid 2:1 htb rate 100mbit ceil 100mbit
tc class add dev eth1 parent 2:1 classid 2:100 htb rate 1mbit ceil 5mbit
tc class add dev eth1 parent 2:1 classid 2:1000 htb rate 1mbit ceil 1mbit

iptables -t mangle -F FORWARD
iptables -t mangle -A FORWARD -o eth0 -d 192.168.1.100 -j CLASSIFY
--set-class 1:100
iptables -t mangle -A FORWARD -o eth1 -s 192.168.1.100 -j CLASSIFY
--set-class 5:100
#not need since class [12]:1000 are already a fetch-all for the unclassified
iptables -t mangle -A OUTPUT -o eth0 -j CLASSIFY --set-class 1:1000
iptables -t mangle -A OUTPUT -o eth1 -j CLASSIFY --set-class 5:1000

That's all.
Of course, a rule reorganization it's a must.

Michele

Re: tc and CONNMARK

[Fabio Marcone]

Hello,
>
>> iptables -t mangle -A PREROUTING -p TCP -m mac --mac-source 
>> xx:xx:xx:xx:xx:xx--dport 443 -j MARK --set-mark 8
>>
>
> Why not forward or postrouting? Do you choose prerouting for use the 
> mac addrs?

yes, I recognize workstation by macaddress and workstations are divided 
into groups. Different groups can use different rates related at the 
same IP/protocol/port. So, to limit traffic in upload I need to know if 
a packet is related to a connection or an another one.

>
>> iptables version: 1.4.3.2 kernel version: 2.6.29.3
>>
>> both patched to use IMQ devices.
>
> Haven't you said this!
> In _all_ my installations, I always skip to use imq. I don't know why,
> but it's simple to use that are already included inside the vanilla 
> kernel
I use IMQ for services exported by router (like http, smtp, ecc ecc) so 
I can limit upload and download traffic of a particular service 
available on a particular interface.

>
>>>
>>> Normally I don't use connmark because when I try some time ago to 
>>> use it, I found some "not marked" problems, so I switch to classid.
>>>  Better and cleaner for me.
>> what kind of problems?
>
> ip filter don't match my data. But it's true that I didn't lost a lot of
> time following that solution because... I find CLASSIFY
>
>> What do you mean with "I switch to classid" ?
>>
>
> It's a wrong definition, sorry. The right one it CLASSIFY!
>
> Simple example, assuming that you have 192.168.1.0/24, need to limit 
> at 5mb
> the ip .100 on both sides on flow and all the others goes at 1mbit all
> together. eth0 lan, eth1 are wan.
>
> tc qdisc add dev eth0 root handle 1: htb default 1000
> tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit ceil 100mbit
> tc class add dev eth0 parent 1:1 classid 1:100 htb rate 1mbit ceil 5mbit
> tc class add dev eth0 parent 1:1 classid 1:1000 htb rate 1mbit ceil 1mbit
>
> tc qdisc add dev eth1 root handle 2: htb default 1000
> tc class add dev eth1 parent 2: classid 2:1 htb rate 100mbit ceil 100mbit
> tc class add dev eth1 parent 2:1 classid 2:100 htb rate 1mbit ceil 5mbit
> tc class add dev eth1 parent 2:1 classid 2:1000 htb rate 1mbit ceil 1mbit
>
> iptables -t mangle -F FORWARD
> iptables -t mangle -A FORWARD -o eth0 -d 192.168.1.100 -j CLASSIFY
> --set-class 1:100
> iptables -t mangle -A FORWARD -o eth1 -s 192.168.1.100 -j CLASSIFY
> --set-class 5:100
> #not need since class [12]:1000 are already a fetch-all for the 
> unclassified
> iptables -t mangle -A OUTPUT -o eth0 -j CLASSIFY --set-class 1:1000
> iptables -t mangle -A OUTPUT -o eth1 -j CLASSIFY --set-class 5:1000
Thanks a lot I didn't known CLASSIFY.

Fabio

Re: tc and CONNMARK [SOLVED]

[Fabio Marcone]

Thanks to all for your help.
>
> Starting a single connection (upload or download) traffic is limited 
> correctly but if I start two connection (one in upload and the other 
> in download) I note that the one in upload runs correctly (how in the 
> previous test) but the one in download stalls and then send some 
> packets and then stalls alternatively.

I solve using sfq to htb leaves, in this way this two flows share the 
bandwidth correctly.

Fabio