Re: [LARTC] Help!!! Bandwith Control with a NAT machine

[Andy Furniss <andy.furniss@dsl.pipex.com>]

Miguel Ángel Domínguez Durán wrote:
> Hello everyone, 
> First of all, sorry for my poor english.
> I've been working with this for a few weeks and I'm getting sick...
> I'm trying to control the bandwith in my network using the following script. The machine where the script is running makes NAT, eth0 is connected to the router and eth1 is connected to the Lan. When I run the script it doesn't appear any errors, i have recompiled a Red Hat kernel 2.4.20, check all the options right and installed iproute2-2.6.9. The result is that every packet is sent to the default queue and I can't understand why. It seems like iptables is not marking any of the packets, all the queues and classes are empty, traffic always goes through default queues in uplink and downlink.
> Here is the script, which is a modification of some things i've found in the net: 
> 
> #!/bin/bash
> #
> #
> 
> DEV1=eth1 #salida a red local
> DEV0=eth0 #salida a internet
> 
> 
> #
> 
> TC=/usr/sbin/tc
> 
> if [ "$1" = "status" ]
> then
>         echo "Enlace descendente"
>         echo "[qdisc]"
>         $TC -s qdisc show dev $DEV1
>         echo "[class]"
>         $TC -s class show dev $DEV1
>         echo "[filter]"
>         $TC -s filter show dev $DEV1
> 
> 
>         echo "Enlace ascendente"
>         echo "[qdisc]"
>         $TC -s qdisc show dev $DEV0
>         echo "[class]"
>         $TC -s class show dev $DEV0
>         echo "[filter]"
>         $TC -s filter show dev $DEV0
> 
> #       echo "[iptables]"
> #       iptables -t mangle -L MYSHAPER-OUT -v -x 2> /dev/null
> #       iptables -t mangle -L MYSHAPER-IN -v -x 2> /dev/null
> 
> 
>         exit
> fi
> 
> # Reset everything to a known state (cleared)
> $TC qdisc del dev $DEV0 root    2> /dev/null > /dev/null
> $TC qdisc del dev $DEV1 root    2> /dev/null > /dev/null
> iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-OUT 2> /dev/null > /dev/null
> iptables -t mangle -F MYSHAPER-OUT 2> /dev/null > /dev/null
> iptables -t mangle -X MYSHAPER-OUT 2> /dev/null > /dev/null
> iptables -t mangle -D PREROUTING -i $DEV1 -j MYSHAPER-IN 2> /dev/null > /dev/null
> iptables -t mangle -F MYSHAPER-IN 2> /dev/null > /dev/null
> iptables -t mangle -X MYSHAPER-IN 2> /dev/null > /dev/null
> 
> #iptables -t mangle -D PREROUTING -i $DEV0 -j MYSHAPER-IN 2> /dev/null > /dev/null
> 
> 
> if [ "$1" = "stop" ]
> then
>         echo "Shaping removed on $DEV1."
>         echo "Shaping removed on $DEV0."
>         exit
> fi
> 
> ###########################################################
> #
> # Inbound Shaping (limits total bandwidth to 1000Kbps)

If you have 1mbit up and down you need to back off a bit from this 
(ceils) - upstream to allow for link overheads - how much depending on 
type of link. Downstream depends on how much you care about latency, as 
a start say 15-20%, you need to do this to have a queue at all.

> # Este es el enlace descendente, desde internet hacia la red interna de Cherrytel
> 
> # set queue size to give latency of about 2 seconds on low-prio packets
> ip link set dev $DEV1 qlen 30

Makes no difference - if you use sfq you can change a define in the 
source or use esfq and specify.

> 
> # changes mtu on the outbound device.  Lowering the mtu will result
> # in lower latency but will also cause slightly lower throughput due
> # to IP and TCP protocol overhead.
> ip link set dev $DEV1 mtu 1000

If I had 1meg symmetrical I doubt I would bother - If you really care 
that much about latency there are other things to do first. If you do 
run low MTU I would specify it as quantum for htb and sfq aswell.

> 
> # add HTB root qdisc
> $TC qdisc add dev $DEV1 root handle 1: htb default 37
> 
> # add main rate limit classes
> $TC class add dev $DEV1 parent 1: classid 1:1 htb rate 1000kbit
> 
> # add leaf classes - We grant each class at LEAST it's "fair share" of bandwidth.
> #                    this way no class will ever be starved by another class.  Each
> #                    class is also permitted to consume all of the available bandwidth
> #                    if no other classes are in use.
> $TC class add dev $DEV1 parent 1:1 classid 1:20 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV1 parent 1:1 classid 1:21 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV1 parent 1:1 classid 1:22 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV1 parent 1:1 classid 1:37 htb rate 832kbit ceil 1000kbit  #por defecto
> 
> $TC class add dev $DEV1 parent 1:1 classid 1:23 htb rate 64kbit ceil 64kbit     #prueba, maq WiFi
> 
> # attach qdisc to leaf classes - here we at SFQ to each priority class.  SFQ insures that
> #                                within each class connections will be treated (almost) fairly.
> $TC qdisc add dev $DEV1 parent 1:20 handle 20: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:21 handle 21: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:22 handle 22: sfq perturb 10
> $TC qdisc add dev $DEV1 parent 1:37 handle 37: sfq perturb 10
> 
> $TC qdisc add dev $DEV1 parent 1:23 handle 23: sfq perturb 10
> 
> # filter traffic into classes by fwmark - here we direct traffic into priority class according to
> #                                         the fwmark set on the packet (we set fwmark with iptables
> #                                         later).  Note that above we've set the default priority
> #                                         class to 1:37 so unmarked packets (or packets marked with
> #                                         unfamiliar IDs) will be defaulted to the lowest priority
> #                                         class.
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 21 fw flowid 1:21
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 22 fw flowid 1:22
> $TC filter add dev $DEV1 parent 1:0 prio 0 protocol ip handle 23 fw flowid 1:23
> 
> # Marking the packets. Se marcan los paquetes en el interfaz contrario, para que no se vean
> # afectados por el NAT que hacen las reglas del firewall
> 
> iptables -t mangle -N MYSHAPER-OUT
> iptables -t mangle -I PREROUTING -i $DEV0 -j MYSHAPER-OUT
> 
> #iptables -t mangle -A MYSHAPER-IN -p tcp --sport ssh -j MARK --set-mark 20
> 
> 
> iptables -A MYSHAPER-OUT -d 172.9.264.30 -t mangle -j MARK --set-mark 20
> iptables -A MYSHAPER-OUT -d 172.9.264.31 -t mangle -j MARK --set-mark 20
> iptables -A MYSHAPER-OUT -d 172.9.264.32 -t mangle -j MARK --set-mark 20
> 
> iptables -A MYSHAPER-OUT -d 172.9.234.22 -t mangle -j MARK --set-mark 21
> iptables -A MYSHAPER-OUT -d 172.9.234.71 -t mangle -j MARK --set-mark 21
> 
> iptables -A MYSHAPER-OUT -d 172.9.234.25 -t mangle -j MARK --set-mark 22
> 
> iptables -A MYSHAPER-OUT -d 172.9.234.14 -t mangle -j MARK --set-mark 23
> 
> # redundant- mark any unmarked packets as 26 (low prio)

This won't mark local adresses as the mangle table in PREROUTING is 
before de-nat happens. Also I thought 172.x.x.x private range started at 
172.16.x.x .

You could move MYSHAPER_OUT (though I would call it IN) to FORWARD or 
use tc filters to match the addresses directly rather than match marks.

> 
> #El resto de tráco iríal flujo por defecto, el 2:37.
> 
> # Done with inbound shaping
> #
> ####################################################
> 
> echo "Control del enlace descendente activado."
> 
> #Si solo se desea controlar el enlace descendente, quitar el comentario de la siguiente instruccion exit
> #exit
> 
> ###########################################################
> #
> # Outbound Shaping (limits total bandwidth to 1000Kbps)
> # Este es el enlace ascendente, desde la red interna de Cherrytel a internet
> 
> # set queue size to give latency of about 2 seconds on low-prio packets
> ip link set dev $DEV0 qlen 30
> 
> # changes mtu on the outbound device.  Lowering the mtu will result
> # in lower latency but will also cause slightly lower throughput due
> # to IP and TCP protocol overhead.
> ip link set dev $DEV0 mtu 1000
> 
> # add HTB root qdisc
> $TC qdisc add dev $DEV0 root handle 2: htb default 73
> 
> # add main rate limit classes
> $TC class add dev $DEV0 parent 2: classid 2:1 htb rate 1000kbit
> 
> # add leaf classes - We grant each class at LEAST it's "fair share" of bandwidth.
> #                    this way no class will ever be starved by another class.  Each
> #                    class is also permitted to consume all of the available bandwidth
> #                    if no other classes are in use.
> $TC class add dev $DEV0 parent 2:1 classid 2:70 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV0 parent 2:1 classid 2:71 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV0 parent 2:1 classid 2:72 htb rate 64kbit ceil 1000kbit   
> $TC class add dev $DEV0 parent 2:1 classid 2:87 htb rate 744kbit ceil 1000kbit 
> 
> $TC class add dev $DEV0 parent 2:1 classid 2:73 htb rate 64kbit ceil 64kbit     #prueba
> 
> # attach qdisc to leaf classes - here we at SFQ to each priority class.  SFQ insures that
> #                                within each class connections will be treated (almost) fairly.
> $TC qdisc add dev $DEV0 parent 2:70 handle 70: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:71 handle 71: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:72 handle 72: sfq perturb 10
> $TC qdisc add dev $DEV0 parent 2:87 handle 87: sfq perturb 10
> 
> $TC qdisc add dev $DEV0 parent 2:73 handle 73: sfq perturb 10
> 
> # filter traffic into classes by fwmark - here we direct traffic into priority class according to
> #                                         the fwmark set on the packet (we set fwmark with iptables
> #                                         later).  Note that above we've set the default priority
> #                                         class to 1:87 so unmarked packets (or packets marked with
> #                                         unfamiliar IDs) will be defaulted to the lowest priority
> #                                         class.
> $TC filter add dev $DEV0 parent 2:0 prio 1 protocol ip handle 70 fw flowid 1:70
> $TC filter add dev $DEV0 parent 2:0 prio 2 protocol ip handle 71 fw flowid 1:71
> $TC filter add dev $DEV0 parent 2:0 prio 3 protocol ip handle 72 fw flowid 1:72
> $TC filter add dev $DEV0 parent 2:0 prio 4 protocol ip handle 73 fw flowid 1:73

These should be flowid 2:70 not 1:70 etc.

Andy.

> 
> # Marking the packets. Se marcan los paquetes en el interfaz contrario, para que no se vean
> # afectados por el NAT que hacen las reglas del firewall
> 
> iptables -t mangle -N MYSHAPER-IN
> iptables -t mangle -I PREROUTING -i $DEV1 -j MYSHAPER-IN
> 
> #iptables -t mangle -A MYSHAPER-IN -p ! tcp -j MARK --set-mark 20
> 
> 
> iptables -A MYSHAPER-IN -s 172.9.234.30 -t mangle -j MARK --set-mark 70
> iptables -A MYSHAPER-IN -s 172.9.234.31 -t mangle -j MARK --set-mark 70
> iptables -A MYSHAPER-IN -s 172.9.234.32 -t mangle -j MARK --set-mark 70
> 
> iptables -A MYSHAPER-IN -s 172.9.234.22 -t mangle -j MARK --set-mark 71
> iptables -A MYSHAPER-IN -s 172.9.234.71 -t mangle -j MARK --set-mark 71
> 
> iptables -A MYSHAPER-IN -s 172.9.234.25 -t mangle -j MARK --set-mark 72
> 
> #Prueba maquina WiFi
> iptables -A MYSHAPER-IN -s 172.9.234.14 -t mangle -j MARK --set-mark 73
> 
> #El resto de tráco iríal flujo por defecto, el 2:87.
> 
> 
> # Done with outbound shaping
> 
> ####################################################
> 
> echo "Control del enlace ascendente activado."
> 
> exit
> 
> Thanks for your help!
> 
> 
> 
> UN CORDIAL SALUDO
> 
> Miguel Ángel Domínguez Durán.
> Departamento Técnico.
> Cherrytel Comunicaciones, S.L.
> mdominguez@cherrytel.com
> http://www.cherrytel.com/
> Tlf. 902 115 673
> Fax 952218170


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/