From mboxrd@z Thu Jan  1 00:00:00 1970
From: Andy Furniss <andy.furniss@dsl.pipex.com>
Date: Fri, 11 Feb 2005 21:42:59 +0000
Subject: Re: [LARTC] Help!!! Bandwith Control with a NAT machine
Message-Id: <420D26E3.8010705@dsl.pipex.com>
List-Id: <lartc.vger.kernel.org>
References: <000e01c51039$23d37c90$0eea090a@PORTATILTEC>
In-Reply-To: <000e01c51039$23d37c90$0eea090a@PORTATILTEC>
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
To: lartc@vger.kernel.org

Miguel =C1ngel Dom=EDnguez Dur=E1n wrote:
> Hello everyone,=20
> First of all, sorry for my poor english.
> I've been working with this for a few weeks and I'm getting sick...
> I'm trying to control the bandwith in my network using the following scri=
pt. The machine where the script is running makes NAT, eth0 is connected to=
 the router and eth1 is connected to the Lan. When I run the script it does=
n't appear any errors, i have recompiled a Red Hat kernel 2.4.20, check all=
 the options right and installed iproute2-2.6.9. The result is that every p=
acket is sent to the default queue and I can't understand why. It seems lik=
e iptables is not marking any of the packets, all the queues and classes ar=
e empty, traffic always goes through default queues in uplink and downlink.
> Here is the script, which is a modification of some things i've found in =
the net:=20
>=20
> #!/bin/bash
> #
> #
>=20
> DEV1=3Deth1 #salida a red local
> DEV0=3Deth0 #salida a internet
>=20
>=20
> #
>=20
> TC=3D/usr/sbin/tc
>=20
> if [ "$1" =3D "status" ]
> then
>         echo "Enlace descendente"
>         echo "[qdisc]"
>         $TC -s qdisc show dev $DEV1
>         echo "[class]"
>         $TC -s class show dev $DEV1
>         echo "[filter]"
>         $TC -s filter show dev $DEV1
>=20
>=20
>         echo "Enlace ascendente"
>         echo "[qdisc]"
>         $TC -s qdisc show dev $DEV0
>         echo "[class]"
>         $TC -s class show dev $DEV0
>         echo "[filter]"
>         $TC -s filter show dev $DEV0
>=20
> #       echo "[iptables]"
> #       iptables -t mangle -L MYSHAPER-OUT -v -x 2> /dev/null
> #       iptables -t mangle -L MYSHAPER-IN -v -x 2> /dev/null
>=20
>=20
>         exit
> fi
>=20
> # Reset everything to a known state (cleared)
> $TC qdisc del dev $DEV0 root    2> /dev/null > /dev/null
> $TC qdisc del dev $DEV1 root    2> /dev/null > /dev/null
> iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-OUT 2> /dev/null > =
/dev/null
> iptables -t mangle -F MYSHAPER-OUT 2> /dev/null > /dev/null
> iptables -t mangle -X MYSHAPER-OUT 2> /dev/null > /dev/null
> iptables -t mangle -D PREROUTING -i $DEV1 -j MYSHAPER-IN 2> /dev/null > /=
dev/null
> iptables -t mangle -F MYSHAPER-IN 2> /dev/null > /dev/null
> iptables -t mangle -X MYSHAPER-IN 2> /dev/null > /dev/null
>=20
> #iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-IN 2> /dev/null > =
/dev/null
>=20
>=20
> if [ "$1" =3D "stop" ]
> then
>         echo "Shaping removed on $DEV1."
>         echo "Shaping removed on $DEV0."
>         exit
> fi
>=20
> ###########################################################
> #
> # Inbound Shaping (limits total bandwidth to 1000Kbps)

If you have 1mbit up and down you need to back off a bit from this=20
(ceils) - upstream to allow for link overheads - how much depending on=20
type of link. Downstream depends on how much you care about latency, as=20
a start say 15-20%, you need to do this to have a queue at all.

> # Este es el enlace descendente, desde internet hacia la red interna de C=
herrytel
>=20
> # set queue size to give latency of about 2 seconds on low-prio packets
> ip link set dev $DEV1 qlen 30

Makes no difference - if you use sfq you can change a define in the=20
source or use esfq and specify.

>=20
> # changes mtu on the outbound device.  Lowering the mtu will result
> # in lower latency but will also cause slightly lower throughput due
> # to IP and TCP protocol overhead.
> ip link set dev $DEV1 mtu 1000

If I had 1meg symmetrical I doubt I would bother - If you really care=20
that much about latency there are other things to do first. If you do=20
run low MTU I would specify it as quantum for htb and sfq aswell.

>=20
> # add HTB root qdisc
> $TC qdisc add dev $DEV1 root handle 1: htb default 37
>=20
> # add main rate limit classes
> $TC class add dev $DEV1 parent 1: classid 1:1 htb rate 1000kbit
>=20
> # add leaf classes - We grant each class at LEAST it's "fair share" of ba=
ndwidth.
> #                    this way no class will ever be starved by another cl=
ass.  Each
> #                    class is also permitted to consume all of the availa=
ble bandwidth
> #                    if no other classes are in use.
> $TC class add dev $DEV1 parent 1:1 classid 1:20 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV1 parent 1:1 classid 1:21 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV1 parent 1:1 classid 1:22 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV1 parent 1:1 classid 1:37 htb rate 832kbit ceil 100=
0kbit  #por defecto
>=20
> $TC class add dev $DEV1 parent 1:1 classid 1:23 htb rate 64kbit ceil 64kb=
it     #prueba, maq WiFi
>=20
> # attach qdisc to leaf classes - here we at SFQ to each priority class.  =
SFQ insures that
> #                                within each class connections will be tr=
eated (almost) fairly.
> $TC qdisc add dev $DEV1 parent 1:20 handle 20: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:21 handle 21: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:22 handle 22: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:37 handle 37: sfq perturb 10
>=20
> $TC qdisc add dev $DEV1 parent 1:23 handle 23: sfq perturb 10
>=20
> # filter traffic into classes by fwmark - here we direct traffic into pri=
ority class according to
> #                                         the fwmark set on the packet (w=
e set fwmark with iptables
> #                                         later).  Note that above we've =
set the default priority
> #                                         class to 1:37 so unmarked packe=
ts (or packets marked with
> #                                         unfamiliar IDs) will be default=
ed to the lowest priority
> #                                         class.
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 20 fw flowi=
d 1:20
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 21 fw flowi=
d 1:21
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 22 fw flowi=
d 1:22
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 23 fw flowi=
d 1:23
>=20
> # Marking the packets. Se marcan los paquetes en el interfaz contrario, p=
ara que no se vean
> # afectados por el NAT que hacen las reglas del firewall
>=20
> iptables -t mangle -N MYSHAPER-OUT
> iptables -t mangle -I PREROUTING -i $DEV0 -j MYSHAPER-OUT
>=20
> #iptables -t mangle -A MYSHAPER-IN -p tcp --sport ssh -j MARK --set-mark =
20
>=20
>=20
> iptables -A MYSHAPER-OUT -d 172.9.264.30 -t mangle -j MARK --set-mark 20
> iptables -A MYSHAPER-OUT -d 172.9.264.31 -t mangle -j MARK --set-mark 20
> iptables -A MYSHAPER-OUT -d 172.9.264.32 -t mangle -j MARK --set-mark 20
>=20
> iptables -A MYSHAPER-OUT -d 172.9.234.22 -t mangle -j MARK --set-mark 21
> iptables -A MYSHAPER-OUT -d 172.9.234.71 -t mangle -j MARK --set-mark 21
>=20
> iptables -A MYSHAPER-OUT -d 172.9.234.25 -t mangle -j MARK --set-mark 22
>=20
> iptables -A MYSHAPER-OUT -d 172.9.234.14 -t mangle -j MARK --set-mark 23
>=20
> # redundant- mark any unmarked packets as 26 (low prio)

This won't mark local adresses as the mangle table in PREROUTING is=20
before de-nat happens. Also I thought 172.x.x.x private range started at=20
172.16.x.x .

You could move MYSHAPER_OUT (though I would call it IN) to FORWARD or=20
use tc filters to match the addresses directly rather than match marks.

>=20
> #El resto de tr=C3=A1co ir=C3=ADal flujo por defecto, el 2:37.
>=20
> # Done with inbound shaping
> #
> ####################################################
>=20
> echo "Control del enlace descendente activado."
>=20
> #Si solo se desea controlar el enlace descendente, quitar el comentario d=
e la siguiente instruccion exit
> #exit
>=20
> ###########################################################
> #
> # Outbound Shaping (limits total bandwidth to 1000Kbps)
> # Este es el enlace ascendente, desde la red interna de Cherrytel a inter=
net
>=20
> # set queue size to give latency of about 2 seconds on low-prio packets
> ip link set dev $DEV0 qlen 30
>=20
> # changes mtu on the outbound device.  Lowering the mtu will result
> # in lower latency but will also cause slightly lower throughput due
> # to IP and TCP protocol overhead.
> ip link set dev $DEV0 mtu 1000
>=20
> # add HTB root qdisc
> $TC qdisc add dev $DEV0 root handle 2: htb default 73
>=20
> # add main rate limit classes
> $TC class add dev $DEV0 parent 2: classid 2:1 htb rate 1000kbit
>=20
> # add leaf classes - We grant each class at LEAST it's "fair share" of ba=
ndwidth.
> #                    this way no class will ever be starved by another cl=
ass.  Each
> #                    class is also permitted to consume all of the availa=
ble bandwidth
> #                    if no other classes are in use.
> $TC class add dev $DEV0 parent 2:1 classid 2:70 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV0 parent 2:1 classid 2:71 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV0 parent 2:1 classid 2:72 htb rate 64kbit ceil 1000=
kbit  =20
> $TC class add dev $DEV0 parent 2:1 classid 2:87 htb rate 744kbit ceil 100=
0kbit=20
>=20
> $TC class add dev $DEV0 parent 2:1 classid 2:73 htb rate 64kbit ceil 64kb=
it     #prueba
>=20
> # attach qdisc to leaf classes - here we at SFQ to each priority class.  =
SFQ insures that
> #                                within each class connections will be tr=
eated (almost) fairly.
> $TC qdisc add dev $DEV0 parent 2:70 handle 70: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:71 handle 71: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:72 handle 72: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:87 handle 87: sfq perturb 10
>=20
> $TC qdisc add dev $DEV0 parent 2:73 handle 73: sfq perturb 10
>=20
> # filter traffic into classes by fwmark - here we direct traffic into pri=
ority class according to
> #                                         the fwmark set on the packet (w=
e set fwmark with iptables
> #                                         later).  Note that above we've =
set the default priority
> #                                         class to 1:87 so unmarked packe=
ts (or packets marked with
> #                                         unfamiliar IDs) will be default=
ed to the lowest priority
> #                                         class.
> $TC filter add dev $DEV0 parent 2:0 prio 1 protocol ip handle 70 fw flowi=
d 1:70
> $TC filter add dev $DEV0 parent 2:0 prio 2 protocol ip handle 71 fw flowi=
d 1:71
> $TC filter add dev $DEV0 parent 2:0 prio 3 protocol ip handle 72 fw flowi=
d 1:72
> $TC filter add dev $DEV0 parent 2:0 prio 4 protocol ip handle 73 fw flowi=
d 1:73

These should be flowid 2:70 not 1:70 etc.

Andy.

>=20
> # Marking the packets. Se marcan los paquetes en el interfaz contrario, p=
ara que no se vean
> # afectados por el NAT que hacen las reglas del firewall
>=20
> iptables -t mangle -N MYSHAPER-IN
> iptables -t mangle -I PREROUTING -i $DEV1 -j MYSHAPER-IN
>=20
> #iptables -t mangle -A MYSHAPER-IN -p ! tcp -j MARK --set-mark 20
>=20
>=20
> iptables -A MYSHAPER-IN -s 172.9.234.30 -t mangle -j MARK --set-mark 70
> iptables -A MYSHAPER-IN -s 172.9.234.31 -t mangle -j MARK --set-mark 70
> iptables -A MYSHAPER-IN -s 172.9.234.32 -t mangle -j MARK --set-mark 70
>=20
> iptables -A MYSHAPER-IN -s 172.9.234.22 -t mangle -j MARK --set-mark 71
> iptables -A MYSHAPER-IN -s 172.9.234.71 -t mangle -j MARK --set-mark 71
>=20
> iptables -A MYSHAPER-IN -s 172.9.234.25 -t mangle -j MARK --set-mark 72
>=20
> #Prueba maquina WiFi
> iptables -A MYSHAPER-IN -s 172.9.234.14 -t mangle -j MARK --set-mark 73
>=20
> #El resto de tr=C3=A1co ir=C3=ADal flujo por defecto, el 2:87.
>=20
>=20
> # Done with outbound shaping
>=20
> ####################################################
>=20
> echo "Control del enlace ascendente activado."
>=20
> exit
>=20
> Thanks for your help!
>=20
>=20
>=20
> UN CORDIAL SALUDO
>=20
> Miguel =C1ngel Dom=EDnguez Dur=E1n.
> Departamento T=E9cnico.
> Cherrytel Comunicaciones, S.L.
> mdominguez@cherrytel.com
> http://www.cherrytel.com/
> Tlf. 902 115 673
> Fax 952218170


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/