From mboxrd@z Thu Jan 1 00:00:00 1970 From: "GoMi ." Date: Tue, 01 Apr 2003 17:53:16 +0000 Subject: [LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why! MIME-Version: 1 Content-Type: multipart/mixed; boundary="----=_NextPart_000_7e24_5678_77b5" Message-Id: List-Id: To: lartc@vger.kernel.org This is a multi-part message in MIME format. ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; charset="iso-8859-1"; format="flowed" Content-Transfer-Encoding: quoted-printable Hello, this is my first message to the board, so please excuse me if i am=20 omiting any prestablished rule ;) Let me explain you my problem.. I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most = of this peer to peer sharing programs. I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for=20 those 2 inet access with a linux firewall separing both routers and my LAN. Right know i have a stateful firewall doing connection tracking (required=20 for load balancing) DROPING all connections but web/smtp/pop3/https/imap=20 etc.... That works fine, but people want to use p2p programs. After doing m= y=20 research, i found out about QoS and ingress/egress queues. My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc..,=20 granting bandwith and leaving the rest to downloads (having these the=20 posibility of ceiling bw as well). I have two egress and two ingress queues attached to the ethernets connecte= d=20 to each of the ethernet connected to the routers. Ingress is done with IMQ, and egress with HTB. At first it seems to work really well, but after half an hour having it=20 working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues,= =20 hence interactive traffic is no interactive anymore.. :) I red somewhere that to decrease latency i had to set the txqueuelen from=20 100 to 30, but still collapses... I have noticed using iptraf, that the number of packets arriving to the=20 LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0= =20 and eth2, is that normal? I attach the configuration files of both the firewall, and the queues... And a little graphical description of my topology: Obviously doing SNAT and LB :) ______ <-------> |Switch| ______ | | | | eth0 <-------> | | |Linux | <----------> Router ADSL1 . | | eth3 | |192.168.3.5 192.168.3.6 . | | <-----> | Box | . | |192.169.1.1| | eth2 . | | | | <----------> Router ADSL2 <------> |______| |______|192.168.4.2 192.168.4.1 I am really desperate, because it has been a month right now, and i am=20 begging to suspect i am a bit of a fool ;) Thank you very much!! _________________________________________________________________ Charla con tus amigos en l=EDnea mediante MSN Messenger:=20 http://messenger.yupimsn.com/ ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; name="stateful firewall.txt"; format=flowed Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="stateful firewall.txt" #killall adslrunning #/usr/local/bin/adslrunning & #/usr/local/bin/adsl/shaper/1 #/usr/local/bin/adsl/shaper/2 ip rule del from 192.168.2.0/24 ip rule del prio 50 ip rule del from 192.168.3.0/24 prio 201 ip rule del from 192.168.4.0/24 prio 202 ip rule del prio 222 ip rule del from 192.168.2.252 ip rule del from 192.168.2.20 ip rule del from 192.168.0.0/24 ip route del default table main ip rule add prio 50 table main ip rule add prio 201 from 192.168.3.0/24 table 201 ip route add default via 192.168.3.3 dev eth2 src 192.168.3.5 proto static table 201 ip route append prohibit default table 201 metric 1 proto static ip rule add prio 202 from 192.168.4.0/24 table 202 ip route add default via 192.168.4.1 dev eth0 src 192.168.4.2 proto static table 202 ip route append prohibit default table 202 metric 1 proto static ip rule add prio 222 table 222 ip route add default table 222 proto static nexthop via 192.168.3.3 dev eth2 nexthop via 192.168.4.1 dev eth0 ################################################## ## Firewall para CMU CHAMINADE ## ## VLAN0=192.168.0.0/255.255.255.0 VLAN1=192.168.1.0/255.255.255.0 VLAN2=192.168.2.0/255.255.255.0 VLAN3=192.168.3.0/255.255.255.0 VLAN4=192.168.4.0/255.255.255.0 ################################################## ## Primero limpiamos todas las reglas ## ## /sbin/iptables -t filter -F /sbin/iptables -t nat -F /sbin/iptables -t mangle -F echo Limpiando reglas... ################################################## ## Aceptamos todas las conexiones## excepto las de nuestra red ## /sbin/iptables -t filter -P FORWARD ACCEPT ################################################## ## SNAT ## /sbin/iptables -t nat -A POSTROUTING -o eth2 -s 192.168.1.0/255.255.0.0 -j SNAT --to 192.168.3.5 /sbin/iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/255.255.0.0 -j SNAT --to 192.168.4.2 ################################################### ## Forzamos a que los usuarios usen el proxy ## ## # /sbin/iptables -t nat -A PREROUTING -s $VLAN2 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080 # /sbin/iptables -t nat -A PREROUTING -s $VLAN0 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080 # /sbin/iptables -t nat -A PREROUTING -s $VLAN1 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080 ################################################### ## No permitimos que utilicen otro proxy ## ## # /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 3128 -j DROP # /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 8080 -j DROP # /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 3128 -j DROP # /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 8080 -j DROP # /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 3128 -j DROP # /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 8080 -j DROP #################################################### ## Stateful Firewall ## ## ## iptables -t filter -N keep_state iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -t filter -A keep_state -j RETURN iptables -t nat -N keep_state iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -t nat -A keep_state -j RETURN iptables -t nat -A PREROUTING -j keep_state iptables -t nat -A POSTROUTING -j keep_state iptables -t nat -A OUTPUT -j keep_state iptables -t filter -A INPUT -j keep_state iptables -t filter -A OUTPUT -j keep_state iptables -t filter -A FORWARD -j keep_state ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; name="ingress-imq-adsl2.txt"; format=flowed Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="ingress-imq-adsl2.txt" DEV=eth2 IDEV=imq1 IQDEV=1 tc qdisc add dev ${IDEV} handle 1: root htb default 1 tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2 tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k prio 1 tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil 1900kbit burst 6k prio 2 tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid 10:1 tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid 10:2 #iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu #iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128 iptables -t mangle -N IMQ${IQDEV} iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -j RETURN iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV} iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV} ip link set ${IDEV} up ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; name="ingress-imq-adsl1.txt"; format=flowed Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="ingress-imq-adsl1.txt" DEV=eth0 IDEV=imq0 IQDEV=0 tc qdisc add dev ${IDEV} handle 1: root htb default 1 tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2 tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k prio 1 tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil 1900kbit burst 6k prio 2 tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid 10:1 tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid 10:2 #iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu #iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128 iptables -t mangle -N IMQ${IQDEV} iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -j RETURN iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV} iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV} ip link set ${IDEV} up ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; name="egress-htb-adsl2.txt"; format=flowed Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="egress-htb-adsl2.txt" DOWNLINK=1800 UPLINK=250 DEV=eth2 #tc qdisc del dev $DEV root #tc qdisc del dev $DEV ingress ##################### ## UPLINK ## tc qdisc add dev $DEV root handle 1: htb default 20 tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil ${UPLINK}kbit tc class add dev $DEV parent 1:1 classid 1:10 htb rate 100kbit ceil 150kbit prio 0 tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil ${UPLINK}kbit prio 1 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10 ##################### ## DOWNLINK ## #tc qdisc add dev $DEV handle ffff: ingress #tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 30k drop flowid :1 ############################ ## Filtros de prioridades ## tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1 0xff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667 0xffff flowid 1:10 ------=_NextPart_000_7e24_5678_77b5 Content-Type: text/plain; name="egress-htb-adsl1.txt"; format=flowed Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="egress-htb-adsl1.txt" DOWNLINK=1800 UPLINK=250 DEV=eth0 #tc qdisc del dev $DEV root #tc qdisc del dev $DEV ingress ##################### ## UPLINK ## tc qdisc add dev $DEV root handle 1: htb default 20 tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil ${UPLINK}kbit tc class add dev $DEV parent 1:1 classid 1:10 htb rate 200kbit ceil 150kbit prio 0 tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil ${UPLINK}kbit prio 1 tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10 ##################### ## DOWNLINK ## #tc qdisc add dev $DEV handle ffff: ingress #tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 30k drop flowid :1 ############################ ## Filtros de prioridades ## tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1 0xff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53 0xffff flowid 1:10 tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667 0xffff flowid 1:10 ------=_NextPart_000_7e24_5678_77b5-- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/