intermittent and unreliable behaviour with iptables scripts

intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1.1: Type: text/plain, Size: 4240 bytes --]

Fellow list users,

I have been attempting to set up a new firewall for my company with the
two goals of 1. implementing statefull packet filtering and 2.
creating a DMZ for servers that need to be accessed by the
outside world, such as HTTP, SMTP, DNS and FTP.

There seems to be a lot to describe here so please stick with me.

I eventually ended up using a script that I ran across at
http://www.linuxguruz.org/iptables/scripts/rc.DMZ.firewall.txt
written by Oskar Andreasson. I was quickly able to reconfigure
the script to match my network configuration and was up and
running. However, I along with my test group of 5 "lucky" users began to see
some
intermittent and unreliable behavior when accessing the internet through
this new firewall most notably when browsing the web. 

When browsing the web, web pages that normally would load very quickly seem 
to hang for an inconsistent amount of time, anywhere between 1 second to 30
seconds or more
before they would even begin to load or would at times never load at all as
if the connection to the web was lost. This state may continue for seemingly
any random amount of time, a few seconds to several minutes or until I
finally
restarted the firewall computer which would bring it around, but would
always
return eventually. Yet users connecting through the current firewall which
is running RedHat
6.2 and using ipfwadm to forward and masquerade experienced none of these
problems. 
I will note that when the firewall is in the state that no web pages will
load, pings to its LAN interface which usually return in about 1ms will time
timeout, but I have not been able to link this to any specific network
issue. 
Nor am I seeing this behaviour anywhere else on our network. 

Also overall speed of our connection seems to be noticeably slower when 
running through this firewall. One example be it a good one or not is 
when running the high speed bandwidth test at http://www.bandwidthplace.com
through the current firewall the average speed returned is between 1.0 and
1.4
Mbps which seems reasonable given that we have 2 T1's that are load balanced
and about 100 users with varying amounts of usage. However, when running the

same test through the new iptables based firewall the average speed returned
usually in the range of 600 to 800 Kbps.

Wondering if this was caused by a bad rule or rules I implemented the
following script 
so there would be no rules. While this is not much of a firewall and would
be
insane to use at all I never experienced any of the problems described above

while using the firewall in this configuration. 
NOTE: my current modification of the script listed above is attached. You
will notice that
all of the DNAT and FORWARDing rules dealing with the DMZ have been
commented out.
They have worked fine but access to and from the DMZ seem to be affected by
the above
mentioned problems as well. 

#!/bin/sh
INET_IP="XXX.XXX.XXX.XXX"
INET_IFACE="eth2"

IPTABLES="/sbin/iptables"
###########################################
# Load all required IPTables modules
#
/sbin/modprobe ip_tables

$IPTABLES -P INPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT
$IPTABLES -P OUTPUT ACCEPT

for i in filter nat mangle
do
$IPTABLES -t $i -F
$IPTABLES -t $i -X
done

#CRITICAL: Enable IP forwarding since it is disabled by default.
echo "1" > /proc/sys/net/ipv4/ip_forward

$IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP
###########################################################################

Finally the last thing to note for now is that I have changed out nearly 
all of the hardware in this box and am currently using the following
components
with RedHat 8.0 and iptables 1.2.7a.

AMD K6-450 processor (REPLACED)
Asus P5A motherboard (REPLACED)
224Mb PC-100 memory (REPLACED)
3 Netgear FA-310TX NICS (REPLACED 3 3Com 905b-TXNMs and 3 3Com 980C-TXs)
1 ATI 8Mb RAGE IIC AGP graphics card (NO X console only)
1 52X Creative Labs IDE CD-ROM (Secondary Master)
1 10Gb IBM 7200Rpm HardDrive (Primary master) (REPLACED)
1 cheap floppy drive 3.5"

If you have read this far you are a saint.
If anyone has any thoughts in general or would care to comment on 
the attached script I would appreciate any advise given.

Thank you,
Doug Watson


[-- Attachment #1.2: Type: text/html, Size: 6866 bytes --]

[-- Attachment #2: iptables --]
[-- Type: application/octet-stream, Size: 7571 bytes --]

#!/bin/sh
#
# rc.DMZ.firewall - DMZ IP Firewall script for Linux 2.4.x and iptables
#
# Copyright (C) 2001  Oskar Andreasson <blueflux@koffein.net>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program or from the site that you downloaded it
# from; if not, write to the Free Software Foundation, Inc., 59 Temple
# Place, Suite 330, Boston, MA  02111-1307   USA
#

###########
# Configuration options, these will speed you up getting this script to
# work with your own setup.

#
# your LAN's IP range and localhost IP. /24 means to only use the first 24 
# bits of the 32 bit IP adress. the same as netmask 255.255.255.0
#
# STATIC_IP is used by me to allow myself to do anything to myself, might
# be a security risc but sometimes I want this. If you don't have a static
# IP, I suggest not using this option at all for now but it's stil
# enabled per default and will add some really nifty security bugs for all
# those who skips reading the documentation=)

LAN_IP="XXX.XXX.XXX.XXX"
LAN_BCAST_ADRESS="XXX.XXX.XXX.XXX"
LAN_IFACE="eth0"

INET_IP="XXX.XXX.XXX.XXX"
INET_IFACE="eth2"

# HTTP_IP=""
# DNS_IP=""
# DMZ_HTTP_IP=""
# DMZ_DNS_IP=""
DMZ_IP="XXX.XXX.XXX.XXX"
DMZ_IFACE="eth1"

LO_IP="127.0.0.1"
LO_IFACE="lo"


IPTABLES="/sbin/iptables"

###########################################
#
# Load all required IPTables modules
#

#
# Needed to initially load modules
#

/sbin/depmod -a

#
# Adds some iptables targets like LOG, REJECT and MASQUARADE.
#

/sbin/modprobe ipt_LOG
/sbin/modprobe ipt_MASQUERADE

#
# Support for connection tracking of FTP and IRC.
#
#/sbin/modprobe ip_conntrack_ftp
#/sbin/modprobe ip_conntrack_irc


#CRITICAL:  Enable IP forwarding since it is disabled by default.
#

echo "1" > /proc/sys/net/ipv4/ip_forward

#
# Dynamic IP users:
#
#echo "1" > /proc/sys/net/ipv4/ip_dynaddr

###########################################
#
# Chain Policies gets set up before any bad packets gets through
#

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP

#
# the allowed chain for TCP connections, utilized in the FORWARD chain
#

$IPTABLES -N allowed
$IPTABLES -A allowed -p TCP --syn -j ACCEPT
$IPTABLES -A allowed -p TCP -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A allowed -p TCP -j DROP

#
# ICMP rules, utilized in the FORWARD chain
#


$IPTABLES -N icmp_packets

# Changed rules totally
$IPTABLES -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j ACCEPT
$IPTABLES -A icmp_packets -p ICMP -s 0/0 --icmp-type 11 -j ACCEPT

#
# bad_tcp_packets chain
#
# Take care of bad TCP packets that we don't want.
#

$IPTABLES -N bad_tcp_packets
$IPTABLES -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG \
--log-prefix "New not syn:"
$IPTABLES -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP

#
# Do some checks for obviously spoofed IP's
#

$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 192.168.0.0/16 -j DROP
$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 10.0.0.0/8 -j DROP
$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 172.16.0.0/12 -j DROP

###########################################
# POSTROUTING chain in the nat table
#
# Enable IP SNAT for all internal networks trying to get out on the Internet
#

$IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP



###########################################
# PREROUTING chain in the nat table
#
# Enable IP Destination NAT for DMZ zone
#

# $IPTABLES -t nat -A PREROUTING -p TCP -i $INET_IFACE -d $HTTP_IP --dport 80 \
# -j DNAT --to-destination $DMZ_HTTP_IP
# $IPTABLES -t nat -A PREROUTING -p TCP -i $INET_IFACE -d $DNS_IP --dport 53 \
# -j DNAT --to-destination $DMZ_DNS_IP
# $IPTABLES -t nat -A PREROUTING -p UDP -i $INET_IFACE -d $DNS_IP --dport 53 \
# -j DNAT --to-destination $DMZ_DNS_IP

###########################################
#
# FORWARD chain
#
#
# Bad TCP packets we don't want
#

$IPTABLES -A FORWARD -p tcp -j bad_tcp_packets


#
# DMZ section
#
# General rules
#

$IPTABLES -A FORWARD -i $DMZ_IFACE -o $INET_IFACE -j ACCEPT
$IPTABLES -A FORWARD -i $INET_IFACE -o $DMZ_IFACE -m state \
--state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $LAN_IFACE -o $DMZ_IFACE -j ACCEPT
$IPTABLES -A FORWARD -i $DMZ_IFACE -o $LAN_IFACE -j ACCEPT

#
# HTTP server
#

# $IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -d $DMZ_HTTP_IP \
# --dport 80 -j allowed
# $IPTABLES -A FORWARD -p ICMP -i $INET_IFACE -o $DMZ_IFACE -d $DMZ_HTTP_IP \
# -j icmp_packets

#
# DNS server
#

# $IPTABLES -A FORWARD -p TCP -i $INET_IFACE -o $DMZ_IFACE -d $DMZ_DNS_IP \
# --dport 53 -j allowed
# $IPTABLES -A FORWARD -p UDP -i $INET_IFACE -o $DMZ_IFACE -d $DMZ_DNS_IP \
# --dport 53 -j ACCEPT
# $IPTABLES -A FORWARD -p ICMP -i $INET_IFACE -o $DMZ_IFACE -d $DMZ_DNS_IP \
# -j icmp_packets

#
# LAN section
#

$IPTABLES -A FORWARD -i $LAN_IFACE -j ACCEPT
$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT 

#
# LOG all packets reaching here
#

$IPTABLES -A FORWARD -m limit --limit 3/minute --limit-burst 3 -j LOG \
--log-level DEBUG --log-prefix "IPT FORWARD packet died: "


###########################################################
#
# Firewall rules
# Rules applying to the firewall box
#
#
# INPUT chain
#
# Bad TCP packets we don't want
#

$IPTABLES -A INPUT -p tcp -j bad_tcp_packets

#
# Packets from the Internet to this box
#

$IPTABLES -A INPUT -p ICMP -i $INET_IFACE -j icmp_packets

#
# Packets from LAN, DMZ or LOCALHOST
#

# From DMZ Interface to DMZ firewall IP
$IPTABLES -A INPUT -p ALL -i $DMZ_IFACE -d $DMZ_IP -j ACCEPT

# From LAN Interface to LAN firewall IP
$IPTABLES -A INPUT -p ALL -i $LAN_IFACE -d $LAN_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -i $LAN_IFACE -d $LAN_BCAST_ADRESS -j ACCEPT

# From Localhost interface to Localhost IP
$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $LO_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $LAN_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $INET_IP -j ACCEPT

# All established and related packets incoming from the internet to the 
# firewall
$IPTABLES -A INPUT -p ALL -d $INET_IP -m state --state ESTABLISHED,RELATED \
-j ACCEPT

# Logging rule
$IPTABLES -A INPUT -m limit --limit 3/minute --limit-burst 3 \
-j LOG --log-level DEBUG --log-prefix "IPT INPUT packet died: "

###########################################################
#
# OUTPUT chain
#
#
# Bad TCP packets we don't want
#

$IPTABLES -A OUTPUT -p tcp -j bad_tcp_packets

#
# Allow ourself to send packets not spoofed everywhere
#

$IPTABLES -A OUTPUT -p ALL -o $LO_IFACE -s $LO_IP -j ACCEPT
$IPTABLES -A OUTPUT -p ALL -o $LAN_IFACE -s $LAN_IP -j ACCEPT
$IPTABLES -A OUTPUT -p ALL -o $INET_IFACE -s $INET_IP -j ACCEPT

#
# Logging rule 
#

$IPTABLES -A OUTPUT -m limit --limit 3/minute --limit-burst 3 -j LOG \
--log-level DEBUG --log-prefix "IPT OUTPUT packet died: "

Re: intermittent and unreliable behaviour with iptables scripts

[alex]

On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
>  However, I along with my test group of 5 "lucky" users began to see
> some
> intermittent and unreliable behavior when accessing the internet
> through
> this new firewall most notably when browsing the web. 
> 
> When browsing the web, web pages that normally would load very quickly
> seem 
> to hang for an inconsistent amount of time, anywhere between 1 second
> to 30 seconds or more
> before they would even begin to load or would at times never load at
> all as
> if the connection to the web was lost.

This sound familiar to my own woes with port forwarded connections. I
suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
end up in the output chains. I've been trying to find out exactly when
this occurs and why (and certainly why my older script worked without
problems).

You could try a using a variation of this script to monitor your
connections "live" and see which rule starts dropping when you
experience your problems. Try using it with something like watch:

    iptables -Z -t nat
    iptables -Z
    watch -n 5 -d ./dumpview

#!/bin/bash
#
# dumpview - try and see where the packets get dropped.
#
echo "DNAT Stuff"
iptables -nvL -t nat
echo "Dropped packets of normal chains"
iptables -nvL | egrep "Chain|DROP"
echo "Connections"
cat /proc/net/ip_conntrack | wc -l
echo "Web Connections"
cat /proc/net/ip_conntrack | grep "port=80" 

-- 
alex <alex@bennee.com>
My own hacking haven

Re: intermittent and unreliable behaviour with iptables scripts

[Anders Fugmann]

Doug Watson wrote:
> 
> When browsing the web, web pages that normally would load very quickly seem
> to hang for an inconsistent amount of time, anywhere between 1 second to 
> 30 seconds or more before they would even begin to load or would at times   > ever load at all as if the connection to the web was lost. This 
state may
> continue for seemingly any random amount of time, a few seconds to 
> several minutes or until I finally restarted the firewall computer which
> would bring it around, but would always return eventually. Yet users 
> connecting through the current firewall which is running RedHat 6.2 and using 
> ipfwadm to forward and masquerade experienced none of these problems.
> I will note that when the firewall is in the state that no web pages will
> load, pings to its LAN interface which usually return in about 1ms will 
> time timeout, but I have not been able to link this to any specific network 
> issue. Nor am I seeing this behaviour anywhere else on our network.
Are any lines logged in the firewall? It may be that the connection 
table cannot hold all the entries. Try increasing it:
echo 65535 > /proc/sys/net/ipv4/ip_conntrack_max.

> 
> Also overall speed of our connection seems to be noticeably slower when
> running through this firewall. One example be it a good one or not is
> when running the high speed bandwidth test at http://www.bandwidthplace.com
> through the current firewall the average speed returned is between 1.0 
> and 1.4 Mbps which seems reasonable given that we have 2 T1's that are load 
> balanced and about 100 users with varying amounts of usage. However, when running 
> the same test through the new iptables based firewall the average speed 
> returned usually in the range of 600 to 800 Kbps.
> 
> Wondering if this was caused by a bad rule or rules I implemented the 
> following script so there would be no rules. While this is not much of a 
> firewall and would be insane to use at all I never experienced any of the problems 
> described above while using the firewall in this configuration.
Good debugging. I guess that you have checked that there are no dropped 
packets in the system logs, when using the attached script. If there are 
none, I would suspect hardware to be the bottleneck.

I would recommend you to try and extend the small sctipt and add 
functionality in small steps while watching performance.

> Finally the last thing to note for now is that I have changed out nearly
> all of the hardware in this box and am currently using the following 
> components with RedHat 8.0 and iptables 1.2.7a.
> 
> AMD K6-450 processor (REPLACED)
> Asus P5A motherboard (REPLACED)
Bad choice. The ASUS P5A can only cache up to 128Mb. Try removing some 
of the ram modules, in order to only have 128 MB installed, and see how 
this affects performance. (Note: I'm running a firewall on a 512Mbits 
internet connection on almost identical hardware (P5A, K6-II 300Mhz, 
192Mb ram) with over 300 iptables rules, and see absolutly no degradation)

> 224Mb PC-100 memory (REPLACED)
> 3 Netgear FA-310TX NICS (REPLACED 3 3Com 905b-TXNMs and 3 3Com 980C-TXs)
> 1 ATI 8Mb RAGE IIC AGP graphics card (NO X console only)
> 1 52X Creative Labs IDE CD-ROM (Secondary Master)
> 1 10Gb IBM 7200Rpm HardDrive (Primary master) (REPLACED)
> 1 cheap floppy drive 3.5"
> 

What kernel are you running. I really recomment that you compile your 
own kernel, and minimizing the number of modules nessesary by linking 
them statically into the kernel. Make sure that you optimize for the K6 
architecture.

Also you mention loadbalacing. Is this done by the firewall or by some 
other hardware?

As a general note to the attached script.
The DMZ rules are really crappy, and it is actually not a DMZ.
_Never_ allow packets from the DMZ to the internal network.
(Well one might allow SSH, but the should be all.). Also do not allow 
the machines in the DMZ to have unrestricted access to the internet. 
Limit it to the services and DNS to specific servers.

Hope it helps.

Regards
Anders Fugmann

Re: intermittent and unreliable behaviour with iptables scripts

[Joel Newkirk]

On Monday 11 November 2002 12:25 pm, Doug Watson wrote:
> Fellow list users,
>
> I have been attempting to set up a new firewall for my company with the
> two goals of 1. implementing statefull packet filtering and 2.
> creating a DMZ for servers that need to be accessed by the
> outside world, such as HTTP, SMTP, DNS and FTP.
>
> There seems to be a lot to describe here so please stick with me.

OK, but I'm going to snip liberally... :^)

> When browsing the web, web pages that normally would load very quickly seem
> to hang for an inconsistent amount of time, anywhere between 1 second to 30

> I will note that when the firewall is in the state that no web pages will
> load, pings to its LAN interface which usually return in about 1ms will
> time timeout, but I have not been able to link this to any specific network

So the problem is probably occurring in PREROUTING or routing, since it hits 
both forward and input packets. Try inserting in nat prerouting and filter 
forward and input:

$IPTABLES -t filter -I FORWARD 1 -i $LAN_IFACE -p ICMP -j LOG --log-level 
debug --log-prefix "IPT:SlowPingFront: "
$IPTABLES -t filter -I FORWARD 1 -o $LAN_IFACE -p ICMP -j LOG --log-level 
debug --log-prefix "IPT:SlowPingBack: "

Reproduce your problem and slow ping (-i 10 perhaps, -w 10000 from windows) 
from the lan to the firewall and beyond, and see what picture the log paints.  
If the ping is just dying somewhere you can (hopefully) see it's trail.  If 
it's really slow (several seconds) you've hopefully narrowed it down to WHERE 
it's slowing.

Check /proc/net/ip_conntrack, general memory issues, and top.

> Also overall speed of our connection seems to be noticeably slower when
> running through this firewall. One example be it a good one or not is
> when running the high speed bandwidth test at http://www.bandwidthplace.com
> through the current firewall the average speed returned is between 1.0 and

> same test through the new iptables based firewall the average speed
> returned usually in the range of 600 to 800 Kbps.

This is quite possibly the same thing as above, just in a less intrusive part 
of the cycle...  However, I had a few suggestions on your forward and 
bad_tcp_packets chains.

- disclaimer -
below is largely feng shui firewalling.  After a dozen readings of your 
firewall (cigarette breaks outside :^) it just didn't 'feel' right, and the 
changes below 'improve the flow' and 'feel right' often without any firm 
justification.  (along with some clarity and simplicity)  However, you DID 
ask for thoughts in general...
- /disclaimer -

Move the spoof tests from bad_tcp_packets to their own chain, or just move the 
state tests directly to the start of forward and input.  (Do you need them 
for output?  You certainly don't need to test for -i $INET_IFACE in 
output...)  Call the spoof tests next, to catch both $LAN and $DMZ bound 
spoofs.

You currently accept EST/REL from $INET back to $DMZ, then have tests for new 
connections on that route, then blanket EST/REL.  Make your unrestricted 
EST/REL rule next instead, followed by your $LAN accept rule, unless you want 
to block otherwise valid EST/REL involving the DMZ, in which case the 
following two rules have to come before the EST/REL.  -i $LAN -o $DMZ ACCEPT 
is redundant, picked up in the -i $LAN ACCEPT that follows in your current 
sequence, and EST/REL -i $INET -o $DMZ or -o $LAN are subsumed in the blanket 
accept.

Follow this up with a jump for -o $DMZ_IFACE -j DMZin, then  -o $DMZ_IFACE -j 
DMZout.  and move all appropriate rules to those chains.  Now you can treat 
DMZin almost like INPUT for the DMZ, and DMZout like OUTPUT, and keep all DMZ 
rules separate from lan<->inet.  You can easily insert/remove a blanket 
accept/reject/drop/return target at the head, or add 'policy' at the end.  
All that these rules need to test now is protocol related, the other 3 
matches for each -i $INET rule are eliminated.  (unless you later use -d or 
-p to filter various DMZ forwards differently)  Also, if you have this before 
your EST/REL then the DMZ chains should each get their own EST/REL at their 
end, before any 'policy' rule.

Finish out with your log rule.  Don't forget to change INPUT (and OUTPUT) to 
reflect the removal of bad_tcp_packets, if you integrated the state rules.

Thus ends today's session of Feng Shui Firewalling.  :^)  The resulting script 
is simpler and more easily modified than currently, and the process is 
simpler than the instructions.  As far as I see, it achieves the same 
filtering and forwarding as your current script with potentially far less 
rule tests per packet, but if I've missed something somebody please correct 
me.

j

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Thank you for your prompt response.

If there really is a bug in ip_conntrack
that makes me unfortunately skiddish about 
continuing on with netfilter/iptables as a 
viable solution for my company. Yet it seems 
like many people have implemented this and have
not seen these types of problems.

I have run the script that you sent me several times.
Attached is a sample output from it. I don't believe that
I am seeing anything too strange, but I do have 1 question.
in the following line which you will see in the attached file
what does the (policy ACCEPT 4 packets, 284 bytes) mean?
Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)

Is that the total number of packets to traverse the OUTPUT 
chain or it he number of packets ACCEPTED by the policy for the 
OUTPUT chain? Or something else?

Thank you,
Doug Watson

-----Original Message-----
From: alex [mailto:alex@bennee.com]
Sent: Monday, November 11, 2002 6:19 PM
To: Doug Watson
Cc: 'netfilter@lists.netfilter.org'
Subject: Re: intermittent and unreliable behaviour with iptables scripts


On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
>  However, I along with my test group of 5 "lucky" users began to see
> some
> intermittent and unreliable behavior when accessing the internet
> through
> this new firewall most notably when browsing the web. 
> 
> When browsing the web, web pages that normally would load very quickly
> seem 
> to hang for an inconsistent amount of time, anywhere between 1 second
> to 30 seconds or more
> before they would even begin to load or would at times never load at
> all as
> if the connection to the web was lost.

This sound familiar to my own woes with port forwarded connections. I
suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
end up in the output chains. I've been trying to find out exactly when
this occurs and why (and certainly why my older script worked without
problems).

You could try a using a variation of this script to monitor your
connections "live" and see which rule starts dropping when you
experience your problems. Try using it with something like watch:

    iptables -Z -t nat
    iptables -Z
    watch -n 5 -d ./dumpview

#!/bin/bash
#
# dumpview - try and see where the packets get dropped.
#
echo "DNAT Stuff"
iptables -nvL -t nat
echo "Dropped packets of normal chains"
iptables -nvL | egrep "Chain|DROP"
echo "Connections"
cat /proc/net/ip_conntrack | wc -l
echo "Web Connections"
cat /proc/net/ip_conntrack | grep "port=80" 

-- 
alex <alex@bennee.com>
My own hacking haven

[-- Attachment #2: Type: text/html, Size: 4648 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 5968 bytes --]

To follow up my last email the following text is a sample output from 
the script listed below.

DNAT Stuff
Chain PREROUTING (policy ACCEPT 2603 packets, 376K bytes)
 pkts bytes target     prot opt in     out     source
destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination         
  549 27070 SNAT       all  --  *      eth2    0.0.0.0/0
0.0.0.0/0          to:y.y.y.y 

Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
 pkts bytes target     prot opt in     out     source
destination         
Dropped packets of normal chains
Chain INPUT (policy DROP 2392 packets, 361K bytes)
Chain FORWARD (policy DROP 0 packets, 0 bytes)
Chain OUTPUT (policy DROP 0 packets, 0 bytes)
Chain allowed (0 references)
    0     0 DROP       tcp  --  *      *       0.0.0.0/0
0.0.0.0/0          
Chain bad_tcp_packets (3 references)
   47 19496 DROP       tcp  --  *      *       0.0.0.0/0
0.0.0.0/0          tcp flags:!0x16/0x02 state NEW 
   27  3478 DROP       all  --  eth2   *       192.168.0.0/16
0.0.0.0/0          
    0     0 DROP       all  --  eth2   *       10.0.0.0/8
0.0.0.0/0          
    0     0 DROP       all  --  eth2   *       172.16.0.0/12
0.0.0.0/0          
Chain icmp_packets (1 references)
Connections
     29
Web Connections
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4918
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4918 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4919
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4919 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4924
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4924 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4925
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4925 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4927
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4927 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4928
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4928 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4929
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4929 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4930
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4930 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4931
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4931 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4932
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4932 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4933
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4933 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=64.236.24.12 sport=4917
dport=80 src=64.236.24.12 dst=y.y.y.y sport=80 dport=4917 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4934
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4934 [ASSURED] use=1 
tcp      6 118 TIME_WAIT src=x.x.x.x dst=207.200.91.248 sport=4920 dport=80
src=207.200.91.248 dst=y.y.y.y sport=80 dport=4920 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.200.91.248 sport=4921
dport=80 src=207.200.91.248 dst=y.y.y.y sport=80 dport=4921 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=207.25.71.252 sport=4936
dport=80 src=207.25.71.252 dst=y.y.y.y sport=80 dport=4936 [ASSURED] use=1 
tcp      6 118 TIME_WAIT src=x.x.x.x dst=207.200.91.248 sport=4922 dport=80
src=207.200.91.248 dst=y.y.y.y sport=80 dport=4922 [ASSURED] use=1 
tcp      6 431999 ESTABLISHED src=x.x.x.x dst=205.188.165.121 sport=4923
dport=80 src=205.188.165.121 dst=y.y.y.y sport=80 dport=4923 [ASSURED] use=1

tcp      6 431999 ESTABLISHED src=x.x.x.x dst=205.188.165.121 sport=4926
dport=80 src=205.188.165.121 dst=y.y.y.y sport=80 dport=4926 [ASSURED] use=1

tcp      6 119 TIME_WAIT src=x.x.x.x dst=63.209.29.151 sport=4935 dport=80
src=63.209.29.151 dst=y.y.y.y sport=80 dport=4935 [ASSURED] use=1 

-----Original Message-----
From: alex [mailto:alex@bennee.com]
Sent: Monday, November 11, 2002 6:19 PM
To: Doug Watson
Cc: 'netfilter@lists.netfilter.org'
Subject: Re: intermittent and unreliable behaviour with iptables scripts


On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
>  However, I along with my test group of 5 "lucky" users began to see
> some
> intermittent and unreliable behavior when accessing the internet
> through
> this new firewall most notably when browsing the web. 
> 
> When browsing the web, web pages that normally would load very quickly
> seem 
> to hang for an inconsistent amount of time, anywhere between 1 second
> to 30 seconds or more
> before they would even begin to load or would at times never load at
> all as
> if the connection to the web was lost.

This sound familiar to my own woes with port forwarded connections. I
suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
end up in the output chains. I've been trying to find out exactly when
this occurs and why (and certainly why my older script worked without
problems).

You could try a using a variation of this script to monitor your
connections "live" and see which rule starts dropping when you
experience your problems. Try using it with something like watch:

    iptables -Z -t nat
    iptables -Z
    watch -n 5 -d ./dumpview

#!/bin/bash
#
# dumpview - try and see where the packets get dropped.
#
echo "DNAT Stuff"
iptables -nvL -t nat
echo "Dropped packets of normal chains"
iptables -nvL | egrep "Chain|DROP"
echo "Connections"
cat /proc/net/ip_conntrack | wc -l
echo "Web Connections"
cat /proc/net/ip_conntrack | grep "port=80" 

-- 
alex <alex@bennee.com>
My own hacking haven

[-- Attachment #2: Type: text/html, Size: 11288 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 5149 bytes --]

Thank you as well for you prompt response.

I have seen lines logged in /var/log/messages
from some of my logging rules prior to a drop but all of these
seem to be packets that should be getting dropped per my rules. 
I have yet to see any entries indicating that /proc/net/ip_conntrack was
full.
However I have increased the value in ip_conntrack_max to 65535 anyway.

Your comments about the Asus P5A are interesting. I will remove some of
the RAM and what that gets me.

As for my kernel it is a vanilla 2.4.18 kernel. Recompiling my own
streamlined
kernel with most or all netfilter modules included is on my to do list
for today or tomorrow. 

The load balancing of the 2 T1s is being done by a Cisco 2621 router.

Finally, your comments on my "DMZ" rules are greatly appreciated and will
help
me finalize my configuration.

Thank you,
Doug Watson

-----Original Message-----
From: Anders Fugmann [mailto:afu@fugmann.dhs.org]
Sent: Monday, November 11, 2002 7:11 PM
To: Doug Watson
Cc: netfilter@lists.netfilter.org
Subject: Re: intermittent and unreliable behaviour with iptables scripts


Doug Watson wrote:
> 
> When browsing the web, web pages that normally would load very quickly
seem
> to hang for an inconsistent amount of time, anywhere between 1 second to 
> 30 seconds or more before they would even begin to load or would at times
> ever load at all as if the connection to the web was lost. This 
state may
> continue for seemingly any random amount of time, a few seconds to 
> several minutes or until I finally restarted the firewall computer which
> would bring it around, but would always return eventually. Yet users 
> connecting through the current firewall which is running RedHat 6.2 and
using 
> ipfwadm to forward and masquerade experienced none of these problems.
> I will note that when the firewall is in the state that no web pages will
> load, pings to its LAN interface which usually return in about 1ms will 
> time timeout, but I have not been able to link this to any specific
network 
> issue. Nor am I seeing this behaviour anywhere else on our network.
Are any lines logged in the firewall? It may be that the connection 
table cannot hold all the entries. Try increasing it:
echo 65535 > /proc/sys/net/ipv4/ip_conntrack_max.

> 
> Also overall speed of our connection seems to be noticeably slower when
> running through this firewall. One example be it a good one or not is
> when running the high speed bandwidth test at
http://www.bandwidthplace.com
> through the current firewall the average speed returned is between 1.0 
> and 1.4 Mbps which seems reasonable given that we have 2 T1's that are
load 
> balanced and about 100 users with varying amounts of usage. However, when
running 
> the same test through the new iptables based firewall the average speed 
> returned usually in the range of 600 to 800 Kbps.
> 
> Wondering if this was caused by a bad rule or rules I implemented the 
> following script so there would be no rules. While this is not much of a 
> firewall and would be insane to use at all I never experienced any of the
problems 
> described above while using the firewall in this configuration.
Good debugging. I guess that you have checked that there are no dropped 
packets in the system logs, when using the attached script. If there are 
none, I would suspect hardware to be the bottleneck.

I would recommend you to try and extend the small sctipt and add 
functionality in small steps while watching performance.

> Finally the last thing to note for now is that I have changed out nearly
> all of the hardware in this box and am currently using the following 
> components with RedHat 8.0 and iptables 1.2.7a.
> 
> AMD K6-450 processor (REPLACED)
> Asus P5A motherboard (REPLACED)
Bad choice. The ASUS P5A can only cache up to 128Mb. Try removing some 
of the ram modules, in order to only have 128 MB installed, and see how 
this affects performance. (Note: I'm running a firewall on a 512Mbits 
internet connection on almost identical hardware (P5A, K6-II 300Mhz, 
192Mb ram) with over 300 iptables rules, and see absolutly no degradation)

> 224Mb PC-100 memory (REPLACED)
> 3 Netgear FA-310TX NICS (REPLACED 3 3Com 905b-TXNMs and 3 3Com 980C-TXs)
> 1 ATI 8Mb RAGE IIC AGP graphics card (NO X console only)
> 1 52X Creative Labs IDE CD-ROM (Secondary Master)
> 1 10Gb IBM 7200Rpm HardDrive (Primary master) (REPLACED)
> 1 cheap floppy drive 3.5"
> 

What kernel are you running. I really recomment that you compile your 
own kernel, and minimizing the number of modules nessesary by linking 
them statically into the kernel. Make sure that you optimize for the K6 
architecture.

Also you mention loadbalacing. Is this done by the firewall or by some 
other hardware?

As a general note to the attached script.
The DMZ rules are really crappy, and it is actually not a DMZ.
_Never_ allow packets from the DMZ to the internal network.
(Well one might allow SSH, but the should be all.). Also do not allow 
the machines in the DMZ to have unrestricted access to the internet. 
Limit it to the services and DNS to specific servers.

Hope it helps.

Regards
Anders Fugmann

[-- Attachment #2: Type: text/html, Size: 8152 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 5788 bytes --]

Thanks for your input.

I unfortunately have had litte chance the last day or two to 
actually put to use the advise given by you and the others who
have sent me feedback due to MEETINGS! 

2 Things you seem to ofer several suggestions regarding reconfiguring
my iptables script that I will attempt to implement and evaluate. I 
may have some questions in a day or so when I attempt to do this.

Also you asked me if I had checked /proc/net/ip_conntrack. What should I be
looking for in ip_conntrack? One thing I did notice in ip_conntrack is
packets
with a destination of xxx.xxx.xxx.255. I believe this to be broadcast
traffic from
some of our workstations. We are working to track this down and stop it. 
What do you see as a possible impact of this broadcast traffic. I have yet
to see that
ip_conntrack has filled up.

Thank you,
Doug Watson
-----Original Message-----
From: Joel Newkirk [mailto:netfilter@newkirk.us]
Sent: Tuesday, November 12, 2002 1:31 AM
To: Doug Watson; 'netfilter@lists.netfilter.org'
Subject: Re: intermittent and unreliable behaviour with iptables scripts


On Monday 11 November 2002 12:25 pm, Doug Watson wrote:
> Fellow list users,
>
> I have been attempting to set up a new firewall for my company with the
> two goals of 1. implementing statefull packet filtering and 2.
> creating a DMZ for servers that need to be accessed by the
> outside world, such as HTTP, SMTP, DNS and FTP.
>
> There seems to be a lot to describe here so please stick with me.

OK, but I'm going to snip liberally... :^)

> When browsing the web, web pages that normally would load very quickly
seem
> to hang for an inconsistent amount of time, anywhere between 1 second to
30

> I will note that when the firewall is in the state that no web pages will
> load, pings to its LAN interface which usually return in about 1ms will
> time timeout, but I have not been able to link this to any specific
network

So the problem is probably occurring in PREROUTING or routing, since it hits

both forward and input packets. Try inserting in nat prerouting and filter 
forward and input:

$IPTABLES -t filter -I FORWARD 1 -i $LAN_IFACE -p ICMP -j LOG --log-level 
debug --log-prefix "IPT:SlowPingFront: "
$IPTABLES -t filter -I FORWARD 1 -o $LAN_IFACE -p ICMP -j LOG --log-level 
debug --log-prefix "IPT:SlowPingBack: "

Reproduce your problem and slow ping (-i 10 perhaps, -w 10000 from windows) 
from the lan to the firewall and beyond, and see what picture the log
paints.  
If the ping is just dying somewhere you can (hopefully) see it's trail.  If 
it's really slow (several seconds) you've hopefully narrowed it down to
WHERE 
it's slowing.

Check /proc/net/ip_conntrack, general memory issues, and top.

> Also overall speed of our connection seems to be noticeably slower when
> running through this firewall. One example be it a good one or not is
> when running the high speed bandwidth test at
http://www.bandwidthplace.com
> through the current firewall the average speed returned is between 1.0 and

> same test through the new iptables based firewall the average speed
> returned usually in the range of 600 to 800 Kbps.

This is quite possibly the same thing as above, just in a less intrusive
part 
of the cycle...  However, I had a few suggestions on your forward and 
bad_tcp_packets chains.

- disclaimer -
below is largely feng shui firewalling.  After a dozen readings of your 
firewall (cigarette breaks outside :^) it just didn't 'feel' right, and the 
changes below 'improve the flow' and 'feel right' often without any firm 
justification.  (along with some clarity and simplicity)  However, you DID 
ask for thoughts in general...
- /disclaimer -

Move the spoof tests from bad_tcp_packets to their own chain, or just move
the 
state tests directly to the start of forward and input.  (Do you need them 
for output?  You certainly don't need to test for -i $INET_IFACE in 
output...)  Call the spoof tests next, to catch both $LAN and $DMZ bound 
spoofs.

You currently accept EST/REL from $INET back to $DMZ, then have tests for
new 
connections on that route, then blanket EST/REL.  Make your unrestricted 
EST/REL rule next instead, followed by your $LAN accept rule, unless you
want 
to block otherwise valid EST/REL involving the DMZ, in which case the 
following two rules have to come before the EST/REL.  -i $LAN -o $DMZ ACCEPT

is redundant, picked up in the -i $LAN ACCEPT that follows in your current 
sequence, and EST/REL -i $INET -o $DMZ or -o $LAN are subsumed in the
blanket 
accept.

Follow this up with a jump for -o $DMZ_IFACE -j DMZin, then  -o $DMZ_IFACE
-j 
DMZout.  and move all appropriate rules to those chains.  Now you can treat 
DMZin almost like INPUT for the DMZ, and DMZout like OUTPUT, and keep all
DMZ 
rules separate from lan<->inet.  You can easily insert/remove a blanket 
accept/reject/drop/return target at the head, or add 'policy' at the end.  
All that these rules need to test now is protocol related, the other 3 
matches for each -i $INET rule are eliminated.  (unless you later use -d or 
-p to filter various DMZ forwards differently)  Also, if you have this
before 
your EST/REL then the DMZ chains should each get their own EST/REL at their 
end, before any 'policy' rule.

Finish out with your log rule.  Don't forget to change INPUT (and OUTPUT) to

reflect the removal of bad_tcp_packets, if you integrated the state rules.

Thus ends today's session of Feng Shui Firewalling.  :^)  The resulting
script 
is simpler and more easily modified than currently, and the process is 
simpler than the instructions.  As far as I see, it achieves the same 
filtering and forwarding as your current script with potentially far less 
rule tests per packet, but if I've missed something somebody please correct 
me.

j

[-- Attachment #2: Type: text/html, Size: 8804 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Raymond Leach]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

On Wed, 2002-11-13 at 16:34, Doug Watson wrote:
> Thank you for your prompt response.
> 
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.
> 
hmm... me neither

> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
> 
It means that the default policy for the OUTPUT chain is to accept
packets and that there have been 4 packets totalling 284 bytes tested
against this chain.

> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?
> 
As above ...

> Thank you,
> Doug Watson
> 
> -----Original Message-----
> From: alex [mailto:alex@bennee.com]
> Sent: Monday, November 11, 2002 6:19 PM
> To: Doug Watson
> Cc: 'netfilter@lists.netfilter.org'
> Subject: Re: intermittent and unreliable behaviour with iptables
> scripts
> 
> 
> On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
> >  However, I along with my test group of 5 "lucky" users began to see
> > some
> > intermittent and unreliable behavior when accessing the internet
> > through
> > this new firewall most notably when browsing the web. 
> > 
> > When browsing the web, web pages that normally would load very
> quickly
> > seem 
> > to hang for an inconsistent amount of time, anywhere between 1
> second
> > to 30 seconds or more
> > before they would even begin to load or would at times never load at
> > all as
> > if the connection to the web was lost.
> 
> This sound familiar to my own woes with port forwarded connections. I
> suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
> end up in the output chains. I've been trying to find out exactly when
> this occurs and why (and certainly why my older script worked without
> problems).
> 
> You could try a using a variation of this script to monitor your
> connections "live" and see which rule starts dropping when you
> experience your problems. Try using it with something like watch:
> 
>     iptables -Z -t nat
>     iptables -Z
>     watch -n 5 -d ./dumpview
> 
> #!/bin/bash
> #
> # dumpview - try and see where the packets get dropped.
> #
> echo "DNAT Stuff"
> iptables -nvL -t nat
> echo "Dropped packets of normal chains"
> iptables -nvL | egrep "Chain|DROP"
> echo "Connections"
> cat /proc/net/ip_conntrack | wc -l
> echo "Web Connections"
> cat /proc/net/ip_conntrack | grep "port=80" 
> 
> -- 
> alex <alex@bennee.com>
> My own hacking haven
-- 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Dax Kelson]

On Wed, 2002-11-13 at 07:34, Doug Watson wrote:
> Thank you for your prompt response.
> 
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.
> 
> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
> 
> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?
> 
> Thank you,
> Doug Watson

I don't think there is a bug, but rather improper rules. This is a
common mistake that beginners and experts make on firewalls.

What you describe:

"When browsing the web, web pages that normally would load very quickly
seem to hang for an inconsistent amount of time, anywhere between 1
second to 30 seconds or more before they would even begin to load or
would at times never load at all as if the connection to the web was
lost."

This is the CLASSIC (bizarre, yes) symptom of IP fragmentation not
working properly. When I first say these symptoms in 1996, it stumped me
for awhile.

Why would IP fragmentation not be working properly? The answer is that
you are not letting ICMP Type 3, Code 4 (ala fragmentation-needed)
packets through.

I don't know what your specific rules are, but generally the solution is
to have the following rule as the first rule in either INPUT, OUTPUT, or
FORWARD (or all three, which one depends on the overall design of your
ruleset).

iptables -A INPUT/OUTPUT/FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

The 'RELATED' will catch those ICMP Typ3, Code 4 packets.

Dax Kelson
Guru Labs

Re: intermittent and unreliable behaviour with iptables scripts

[Joel Newkirk]

On Wednesday 13 November 2002 10:16 am, Raymond Leach wrote:
> On Wed, 2002-11-13 at 16:34, Doug Watson wrote:
> > what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> > Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
>
> It means that the default policy for the OUTPUT chain is to accept
> packets and that there have been 4 packets totalling 284 bytes tested
> against this chain.

Actually, this tells us that 4 packets totalling 284 bytes made it through the 
output chain rules and were ACCEPTED by policy rather than a rule.  If an 
earlier rule in the chain led to an ACCEPT or a DROP or RETURN then it will 
NOT be counted here, but only in the rule(s) it matched.

> > Is that the total number of packets to traverse the OUTPUT
> > chain or it he number of packets ACCEPTED by the policy for the
> > OUTPUT chain? Or something else?
>
> As above ...

j

RE: intermittent and unreliable behaviour with iptables scripts

[alex]

On Wed, 2002-11-13 at 14:34, Doug Watson wrote:
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.

Agreed. I finally solved my problem which was due to me not allowing
ICMP packets to sent out the OUTPUT chain for established connections
that where being forwarded. My bad, *not* a conntrack problem, it just
seemed that way at the time.

> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)

Your attachment seems to of been filtered out. The policy lines mean
that no rules in the chain matched and therefor the default policy took
effect. It is usually recommended to have a default DROP policy if your
being paranoid and explicitly allow the connections you want to come
through. However as I learnt the hard way you have to make sure the
related traffic is also allowed through. The ICMP packet that bit me was
due to fragmentation being required but not possible on the gateway,
probably a symptom of the fact my two interfaces (ppp and eth) had
different MTU's. 

> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?

Just the packets that defaulted to the policy, the other counts will be
against each rule itself.

-- 
Alex
http://www.bennee.com/~alex/

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 4737 bytes --]

Thanks for the response. What you mentioned in your first
paragraph was and may still be part of my problem.

Per the suggestion of another list user I am on the trail
of RELATED packets that may be getting DROPped. I think that 
I have adjusted my rules in the INPUT, OUTPUT and FORWARD
chains properly to allow for this. 

But the latest development in the enigma that surrounds my firewall is this.

For one, I did not know where packets were being logged when the --log-level
option
was set to DEBUG. Can anyone clarify this for me?
 
More importantly I simply removed the --log-level option from my logging 
rule in the FORWARD chain prior to a drop by the FORWARD chains policy
(DROP)
so that they would be logged to /var/log/messages where I could find them.
What I have seen both disturbs and confuses me terribly. 
I am seeing packets that have a source address from my LAN that are destined
to an external network that are hitting the firewall on its WAN interface
(eth2) rather
than its LAN interface (eth0) and being DROPped by the FORWARD chain's
policy. 
I would think that these packets should be incoming on the firewall's LAN
interface.
This seems to coincide with the intermittent and unreliable performance that
I have
been seeing.
For example yesterday I was at the console of the firewall and using
tail -f /var/log/messages to see what was being logged and tcpdump I started
seeing
these packets being logged when someone was attempting to browse the
internet and could not.
Eventually without explanation that person was able to get through to some
web pages and when
this happened I stopped seeing these packets being logged and dropped.

My set up is a tad screwy in that I am trying to test this along side my
current production firewall and
I only have 1 source of bandwidth which comes and goes through a Cisco 2621
router. I have the ethernet
from the Cisco plugged into a 3Com 3300 switch. I also have the current
firewalls external interface
plugged into this switch along with some of our servers. I also have the LAN
and WAN interfaces of the 
firewall in question plugged into this switch. The switch is unlinked via
fiber to the rest of the
network. In the end I would have the ethernet from the Cisco plugged
directly into the WAN interface
of the new (questionable) firewall. 

That being said has anyone seen the situation described above before? Do you
think that my switch or atleast a 
port on the switch could be to blame for these packets arriving on the wrong
interface if
that is what truly is happening.

Thank you,
Doug Watson
Director of Information Systems
1stBooks Library
http://www.1stbooks.com
dwatson@1stbooks.com
1 800 839 8640 Toll Free
1 812 339 6554 Fax


-----Original Message-----
From: alex [mailto:alex@bennee.com]
Sent: Wednesday, November 13, 2002 5:48 PM
To: Doug Watson
Cc: netfilter@lists.netfilter.org
Subject: RE: intermittent and unreliable behaviour with iptables scripts


On Wed, 2002-11-13 at 14:34, Doug Watson wrote:
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.

Agreed. I finally solved my problem which was due to me not allowing
ICMP packets to sent out the OUTPUT chain for established connections
that where being forwarded. My bad, *not* a conntrack problem, it just
seemed that way at the time.

> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)

Your attachment seems to of been filtered out. The policy lines mean
that no rules in the chain matched and therefor the default policy took
effect. It is usually recommended to have a default DROP policy if your
being paranoid and explicitly allow the connections you want to come
through. However as I learnt the hard way you have to make sure the
related traffic is also allowed through. The ICMP packet that bit me was
due to fragmentation being required but not possible on the gateway,
probably a symptom of the fact my two interfaces (ppp and eth) had
different MTU's. 

> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?

Just the packets that defaulted to the policy, the other counts will be
against each rule itself.

-- 
Alex
http://www.bennee.com/~alex/

[-- Attachment #2: Type: text/html, Size: 7314 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 2806 bytes --]

Thanks to everyone that has responded to my numerous submissions.
I think that I finally have my issues resolved which seem to have been
related to
ARP requests being answered by an improper interface due to my switch set up
and the kernel. 

In searching for kernel related ARP articles per Julian using Google I found
the following suggestion.
	echo 1 > /proc/sys/net/ipv4/conf/all/hidden
However this file did not exist on my RedHat 8 box. But I did find
/proc/sys/net/ipv4/conf/arp_filter.
After doing some quick reading I found that echoing a value of 1 to this
file will cause your computer respond to ARP requests only with the
interface that they were received on. This has fixed my problem until I can
re-cable my set up.

Thanks,
Doug Watson
Director of Information Systems
1stBooks Library
http://www.1stbooks.com
dwatson@1stbooks.com
 

-----Original Message-----
From: Julian Gomez [mailto:kluivert@tm.net.my]
Sent: Sunday, December 08, 2002 2:38 AM
To: Doug Watson
Subject: RE: intermittent and unreliable behaviour with iptables scripts


On Fri, 6 Dec 2002, Doug Watson wrote:

(snip)

> At some point my workstation made an arp request to find out what mac
> address was associated with the ip address on the internal interface
> of the firewall, which is my workstations default gateway. The
> firewall may have choosen to respond with it's external interface
> which is plugged

Yup.

(snip)

> on the wrong interface at the firewall such as http requests to the
> internet that should first arrive that the firewall's internal
> interface and then be routed out to the web. Please let me know if I
> have failed to understand this.

Correct. I suggest you separate both the switches if possible though 
(physically). Usually for my uplink ports to the ISPs I just throw in an 
el-cheapo 10mbit hub.
 
> As for your search suggetion I did find some suggestions to basically
> hide an interface to prevent it from responding to arp requests. Here
> is one example that I found, note I have not tried this yet but may
> very soon.  I don't know what other consequences this may have though.
> 
> # Allow hiding interfaces
> echo 1 > /proc/sys/net/ipv4/conf/all/hidden

I've never used the hidden options, but the above proc entries are the 
ones to modify however -- you're right. Its not available in the default 
pristine kernel, you'll need to patch it.

> Again, thank you very much for you help. If I understand this like I
> think I do I guess I understand why my problem is occuring. I just
> don't understand why the kernel would respond to an arp request on any
> interface other than the one that I came in on.

If I remember correctly, its partly to help load balancing. Its a specific
Linux kernel behaviour though, I don't think it happens on BSD variants 
for example.

HTH.

[-- Attachment #2: Type: text/html, Size: 4797 bytes --]