* [LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why!
@ 2003-04-01 17:53 GoMi .
2003-04-02 20:27 ` Stef Coene
0 siblings, 1 reply; 2+ messages in thread
From: GoMi . @ 2003-04-01 17:53 UTC (permalink / raw)
To: lartc
[-- Attachment #1: Type: text/plain, Size: 2553 bytes --]
Hello, this is my first message to the board, so please excuse me if i am
omiting any prestablished rule ;)
Let me explain you my problem..
I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most
of this peer to peer sharing programs.
I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for
those 2 inet access with a linux firewall separing both routers and my LAN.
Right know i have a stateful firewall doing connection tracking (required
for load balancing) DROPING all connections but web/smtp/pop3/https/imap
etc.... That works fine, but people want to use p2p programs. After doing my
research, i found out about QoS and ingress/egress queues.
My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc..,
granting bandwith and leaving the rest to downloads (having these the
posibility of ceiling bw as well).
I have two egress and two ingress queues attached to the ethernets connected
to each of the ethernet connected to the routers.
Ingress is done with IMQ, and egress with HTB.
At first it seems to work really well, but after half an hour having it
working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues,
hence interactive traffic is no interactive anymore.. :)
I red somewhere that to decrease latency i had to set the txqueuelen from
100 to 30, but still collapses...
I have noticed using iptraf, that the number of packets arriving to the
LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0
and eth2, is that normal?
I attach the configuration files of both the firewall, and the queues...
And a little graphical description of my topology:
Obviously doing SNAT and LB :)
______
<hub1> <-------> |Switch| ______
| | | | eth0
<hub2> <-------> | | |Linux | <----------> Router ADSL1
. | | eth3 | |192.168.3.5 192.168.3.6
. | | <-----> | Box |
. | |192.169.1.1| | eth2
. | | | | <----------> Router ADSL2
<hub24> <------> |______| |______|192.168.4.2 192.168.4.1
I am really desperate, because it has been a month right now, and i am
begging to suspect i am a bit of a fool ;) Thank you very much!!
_________________________________________________________________
Charla con tus amigos en línea mediante MSN Messenger:
http://messenger.yupimsn.com/
[-- Attachment #2: stateful firewall.txt --]
[-- Type: text/plain, Size: 3583 bytes --]
#killall adslrunning
#/usr/local/bin/adslrunning &
#/usr/local/bin/adsl/shaper/1
#/usr/local/bin/adsl/shaper/2
ip rule del from 192.168.2.0/24
ip rule del prio 50
ip rule del from 192.168.3.0/24 prio 201
ip rule del from 192.168.4.0/24 prio 202
ip rule del prio 222
ip rule del from 192.168.2.252
ip rule del from 192.168.2.20
ip rule del from 192.168.0.0/24
ip route del default table main
ip rule add prio 50 table main
ip rule add prio 201 from 192.168.3.0/24 table 201
ip route add default via 192.168.3.3 dev eth2 src 192.168.3.5 proto static
table 201
ip route append prohibit default table 201 metric 1 proto static
ip rule add prio 202 from 192.168.4.0/24 table 202
ip route add default via 192.168.4.1 dev eth0 src 192.168.4.2 proto static
table 202
ip route append prohibit default table 202 metric 1 proto static
ip rule add prio 222 table 222
ip route add default table 222 proto static nexthop via 192.168.3.3 dev eth2
nexthop via 192.168.4.1 dev eth0
##################################################
## Firewall para CMU CHAMINADE
##
##
VLAN0=192.168.0.0/255.255.255.0
VLAN1=192.168.1.0/255.255.255.0
VLAN2=192.168.2.0/255.255.255.0
VLAN3=192.168.3.0/255.255.255.0
VLAN4=192.168.4.0/255.255.255.0
##################################################
## Primero limpiamos todas las reglas
##
##
/sbin/iptables -t filter -F
/sbin/iptables -t nat -F
/sbin/iptables -t mangle -F
echo Limpiando reglas...
##################################################
## Aceptamos todas las conexiones## excepto las de nuestra red
##
/sbin/iptables -t filter -P FORWARD ACCEPT
##################################################
## SNAT
##
/sbin/iptables -t nat -A POSTROUTING -o eth2 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.3.5
/sbin/iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.4.2
###################################################
## Forzamos a que los usuarios usen el proxy
##
##
# /sbin/iptables -t nat -A PREROUTING -s $VLAN2 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080
# /sbin/iptables -t nat -A PREROUTING -s $VLAN0 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080
# /sbin/iptables -t nat -A PREROUTING -s $VLAN1 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080
###################################################
## No permitimos que utilicen otro proxy
##
##
# /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
####################################################
## Stateful Firewall
##
##
##
iptables -t filter -N keep_state
iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
iptables -t filter -A keep_state -j RETURN
iptables -t nat -N keep_state
iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
iptables -t nat -A keep_state -j RETURN
iptables -t nat -A PREROUTING -j keep_state
iptables -t nat -A POSTROUTING -j keep_state
iptables -t nat -A OUTPUT -j keep_state
iptables -t filter -A INPUT -j keep_state
iptables -t filter -A OUTPUT -j keep_state
iptables -t filter -A FORWARD -j keep_state
[-- Attachment #3: ingress-imq-adsl2.txt --]
[-- Type: text/plain, Size: 1849 bytes --]
DEV=eth2
IDEV=imq1
IQDEV=1
tc qdisc add dev ${IDEV} handle 1: root htb default 1
tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit
tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k
prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil
1900kbit burst 6k prio 2
tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo
tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq
tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid
10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid
10:2
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
--clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -j RETURN
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV}
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}
ip link set ${IDEV} up
[-- Attachment #4: ingress-imq-adsl1.txt --]
[-- Type: text/plain, Size: 1849 bytes --]
DEV=eth0
IDEV=imq0
IQDEV=0
tc qdisc add dev ${IDEV} handle 1: root htb default 1
tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit
tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k
prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil
1900kbit burst 6k prio 2
tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo
tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq
tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid
10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid
10:2
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
--clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -j RETURN
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV}
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}
ip link set ${IDEV} up
[-- Attachment #5: egress-htb-adsl2.txt --]
[-- Type: text/plain, Size: 1674 bytes --]
DOWNLINK=1800
UPLINK=250
DEV=eth2
#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress
#####################
## UPLINK
##
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil
${UPLINK}kbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 100kbit ceil 150kbit
prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil
${UPLINK}kbit prio 1
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
#####################
## DOWNLINK
##
#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0
police rate ${DOWNLINK}kbit burst 30k drop flowid :1
############################
## Filtros de prioridades
##
tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid
1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1
0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667
0xffff flowid 1:10
[-- Attachment #6: egress-htb-adsl1.txt --]
[-- Type: text/plain, Size: 1674 bytes --]
DOWNLINK=1800
UPLINK=250
DEV=eth0
#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress
#####################
## UPLINK
##
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil
${UPLINK}kbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 200kbit ceil 150kbit
prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil
${UPLINK}kbit prio 1
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
#####################
## DOWNLINK
##
#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0
police rate ${DOWNLINK}kbit burst 30k drop flowid :1
############################
## Filtros de prioridades
##
tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid
1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1
0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667
0xffff flowid 1:10
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why!
2003-04-01 17:53 [LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why! GoMi .
@ 2003-04-02 20:27 ` Stef Coene
0 siblings, 0 replies; 2+ messages in thread
From: Stef Coene @ 2003-04-02 20:27 UTC (permalink / raw)
To: lartc
On Tuesday 01 April 2003 19:53, GoMi . wrote:
> Hello, this is my first message to the board, so please excuse me if i am
> omiting any prestablished rule ;)
>
> Let me explain you my problem..
>
> I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most
> of this peer to peer sharing programs.
>
> I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for
> those 2 inet access with a linux firewall separing both routers and my LAN.
>
> Right know i have a stateful firewall doing connection tracking (required
> for load balancing) DROPING all connections but web/smtp/pop3/https/imap
> etc.... That works fine, but people want to use p2p programs. After doing
> my research, i found out about QoS and ingress/egress queues.
>
> My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc..,
> granting bandwith and leaving the rest to downloads (having these the
> posibility of ceiling bw as well).
>
> I have two egress and two ingress queues attached to the ethernets
> connected to each of the ethernet connected to the routers.
>
> Ingress is done with IMQ, and egress with HTB.
>
> At first it seems to work really well, but after half an hour having it
> working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues,
> hence interactive traffic is no interactive anymore.. :)
>
> I red somewhere that to decrease latency i had to set the txqueuelen from
> 100 to 30, but still collapses...
>
> I have noticed using iptraf, that the number of packets arriving to the
> LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0
> and eth2, is that normal?
>
> I attach the configuration files of both the firewall, and the queues...
>
> And a little graphical description of my topology:
> Obviously doing SNAT and LB :)
> ______
> <hub1> <-------> |Switch| ______
>
> | | | | eth0
>
> <hub2> <-------> | | |Linux | <----------> Router ADSL1
> . | | eth3 | |192.168.3.5 192.168.3.6
> . | | <-----> | Box |
> . | |192.169.1.1| | eth2
> . | | | | <----------> Router ADSL2
> <hub24> <------> |______| |______|192.168.4.2 192.168.4.1
>
> I am really desperate, because it has been a month right now, and i am
> begging to suspect i am a bit of a fool ;) Thank you very much!!
I don't know the solution to you problem. But I have some remarks.
The burst you configured, is too low. If you remove the burst parameter, htb
will calculate the lowest prossible for you.
Also, just as test, give each class the same prio.
Pref (or prio) if you add a filter is only used to determine the order the
filters are checked. In your case, the pref doesn't mather so you take the
same pref for the filters.
And is it each time after the same second/minutes that you have problems with
interactive traffic? And/or does it depends on the load of the link?
Stef
--
stef.coene@docum.org
"Using Linux as bandwidth manager"
http://www.docum.org/
#lartc @ irc.oftc.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2003-04-02 20:27 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-04-01 17:53 [LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why! GoMi .
2003-04-02 20:27 ` Stef Coene
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.