RE: intermittent and unreliable behaviour with iptables scripts

RE: intermittent and unreliable behaviour with iptables scripts

[Doug Watson]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Thank you for your prompt response.

If there really is a bug in ip_conntrack
that makes me unfortunately skiddish about 
continuing on with netfilter/iptables as a 
viable solution for my company. Yet it seems 
like many people have implemented this and have
not seen these types of problems.

I have run the script that you sent me several times.
Attached is a sample output from it. I don't believe that
I am seeing anything too strange, but I do have 1 question.
in the following line which you will see in the attached file
what does the (policy ACCEPT 4 packets, 284 bytes) mean?
Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)

Is that the total number of packets to traverse the OUTPUT 
chain or it he number of packets ACCEPTED by the policy for the 
OUTPUT chain? Or something else?

Thank you,
Doug Watson

-----Original Message-----
From: alex [mailto:alex@bennee.com]
Sent: Monday, November 11, 2002 6:19 PM
To: Doug Watson
Cc: 'netfilter@lists.netfilter.org'
Subject: Re: intermittent and unreliable behaviour with iptables scripts


On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
>  However, I along with my test group of 5 "lucky" users began to see
> some
> intermittent and unreliable behavior when accessing the internet
> through
> this new firewall most notably when browsing the web. 
> 
> When browsing the web, web pages that normally would load very quickly
> seem 
> to hang for an inconsistent amount of time, anywhere between 1 second
> to 30 seconds or more
> before they would even begin to load or would at times never load at
> all as
> if the connection to the web was lost.

This sound familiar to my own woes with port forwarded connections. I
suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
end up in the output chains. I've been trying to find out exactly when
this occurs and why (and certainly why my older script worked without
problems).

You could try a using a variation of this script to monitor your
connections "live" and see which rule starts dropping when you
experience your problems. Try using it with something like watch:

    iptables -Z -t nat
    iptables -Z
    watch -n 5 -d ./dumpview

#!/bin/bash
#
# dumpview - try and see where the packets get dropped.
#
echo "DNAT Stuff"
iptables -nvL -t nat
echo "Dropped packets of normal chains"
iptables -nvL | egrep "Chain|DROP"
echo "Connections"
cat /proc/net/ip_conntrack | wc -l
echo "Web Connections"
cat /proc/net/ip_conntrack | grep "port=80" 

-- 
alex <alex@bennee.com>
My own hacking haven

[-- Attachment #2: Type: text/html, Size: 4648 bytes --]

RE: intermittent and unreliable behaviour with iptables scripts

[Raymond Leach]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

On Wed, 2002-11-13 at 16:34, Doug Watson wrote:
> Thank you for your prompt response.
> 
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.
> 
hmm... me neither

> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
> 
It means that the default policy for the OUTPUT chain is to accept
packets and that there have been 4 packets totalling 284 bytes tested
against this chain.

> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?
> 
As above ...

> Thank you,
> Doug Watson
> 
> -----Original Message-----
> From: alex [mailto:alex@bennee.com]
> Sent: Monday, November 11, 2002 6:19 PM
> To: Doug Watson
> Cc: 'netfilter@lists.netfilter.org'
> Subject: Re: intermittent and unreliable behaviour with iptables
> scripts
> 
> 
> On Mon, 2002-11-11 at 17:25, Doug Watson wrote:
> >  However, I along with my test group of 5 "lucky" users began to see
> > some
> > intermittent and unreliable behavior when accessing the internet
> > through
> > this new firewall most notably when browsing the web. 
> > 
> > When browsing the web, web pages that normally would load very
> quickly
> > seem 
> > to hang for an inconsistent amount of time, anywhere between 1
> second
> > to 30 seconds or more
> > before they would even begin to load or would at times never load at
> > all as
> > if the connection to the web was lost.
> 
> This sound familiar to my own woes with port forwarded connections. I
> suspect a bug in ip_conntrack that somehow causes FORWARDED packets to
> end up in the output chains. I've been trying to find out exactly when
> this occurs and why (and certainly why my older script worked without
> problems).
> 
> You could try a using a variation of this script to monitor your
> connections "live" and see which rule starts dropping when you
> experience your problems. Try using it with something like watch:
> 
>     iptables -Z -t nat
>     iptables -Z
>     watch -n 5 -d ./dumpview
> 
> #!/bin/bash
> #
> # dumpview - try and see where the packets get dropped.
> #
> echo "DNAT Stuff"
> iptables -nvL -t nat
> echo "Dropped packets of normal chains"
> iptables -nvL | egrep "Chain|DROP"
> echo "Connections"
> cat /proc/net/ip_conntrack | wc -l
> echo "Web Connections"
> cat /proc/net/ip_conntrack | grep "port=80" 
> 
> -- 
> alex <alex@bennee.com>
> My own hacking haven
-- 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: intermittent and unreliable behaviour with iptables scripts

[Joel Newkirk]

On Wednesday 13 November 2002 10:16 am, Raymond Leach wrote:
> On Wed, 2002-11-13 at 16:34, Doug Watson wrote:
> > what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> > Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
>
> It means that the default policy for the OUTPUT chain is to accept
> packets and that there have been 4 packets totalling 284 bytes tested
> against this chain.

Actually, this tells us that 4 packets totalling 284 bytes made it through the 
output chain rules and were ACCEPTED by policy rather than a rule.  If an 
earlier rule in the chain led to an ACCEPT or a DROP or RETURN then it will 
NOT be counted here, but only in the rule(s) it matched.

> > Is that the total number of packets to traverse the OUTPUT
> > chain or it he number of packets ACCEPTED by the policy for the
> > OUTPUT chain? Or something else?
>
> As above ...

j

RE: intermittent and unreliable behaviour with iptables scripts

[Dax Kelson]

On Wed, 2002-11-13 at 07:34, Doug Watson wrote:
> Thank you for your prompt response.
> 
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.
> 
> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)
> 
> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?
> 
> Thank you,
> Doug Watson

I don't think there is a bug, but rather improper rules. This is a
common mistake that beginners and experts make on firewalls.

What you describe:

"When browsing the web, web pages that normally would load very quickly
seem to hang for an inconsistent amount of time, anywhere between 1
second to 30 seconds or more before they would even begin to load or
would at times never load at all as if the connection to the web was
lost."

This is the CLASSIC (bizarre, yes) symptom of IP fragmentation not
working properly. When I first say these symptoms in 1996, it stumped me
for awhile.

Why would IP fragmentation not be working properly? The answer is that
you are not letting ICMP Type 3, Code 4 (ala fragmentation-needed)
packets through.

I don't know what your specific rules are, but generally the solution is
to have the following rule as the first rule in either INPUT, OUTPUT, or
FORWARD (or all three, which one depends on the overall design of your
ruleset).

iptables -A INPUT/OUTPUT/FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

The 'RELATED' will catch those ICMP Typ3, Code 4 packets.

Dax Kelson
Guru Labs

ICMP question

[Luis Fernando Barrera]

Hi all,

I have a RedHat 7.3 box which I use as a dual homed firewall.

One of the rules is that I allow all the ICMP traffic from one
PC in the protected LAN to the Internet.

Here it is one set of rules

iptables -A FORWARD -p ICMP -s 192.168.1.37 -m state --state
NEW,ESTABLISHED -j ACCEPT
iptables -A FORWARD -p ICMP -d 192.168.1.37 -m state --state ESTABLISHED -j
ACCEPT

I've been experiencing the problem that with this set of rules, suddenly,
the PC in the LAN cannot "ping" any hosts in the Internet. To solve the
problem
I delete all my user's chains and flush them, and I worked again.
I already asked if there is a buffer or counter that gets filled, and
someone
told me to watch the number of connections being tracked. I did that and it
seems
that the number of connections being tracked is very low (about 160).

Since the problem was a "mistery", I changed the rules to this:

iptables -A FORWARD -p ICMP -s 192.168.1.37 -j ACCEPT
iptables -A FORWARD -p ICMP -d 192.168.1.37 -j ACCEPT

These are the same rules, except that don't track the state of the
connections...

Could someone tell me what is the risk asociated to NOT to track the
connections?

One more question... Is it true that when you specify the ICMP protocol,
iptables only
applies the rule to the type 0?

thanks in advance

Luis Fernando Barrera
luba@assist.com.gt

RE: ICMP question

[Darrell F. Dieringer]

Anyone please feel free to correct me if I'm wrong, but I think your
original rules would work fine if your "ESTABLISHED" rule was changed
to "ESTABLISHED,RELATED".  Perhaps the "NEW,ESTABLISHED" rule should
also say "NEW,ESTABLISHED,RELATED" as well.

If your internal machine issues an "echo request", the "echo reply"
will be a "RELATED" packet.  (I'd use icmp type numbers, but I don't
know them off the top of my head.)

Darrell Dieringer - Madison, WI

> -----Original Message-----
> From: netfilter-admin@lists.netfilter.org
> [mailto:netfilter-admin@lists.netfilter.org]On Behalf Of
> Luis Fernando
> Barrera
> Sent: Thursday, November 14, 2002 12:28 PM
> To: netfilter@lists.netfilter.org
> Subject: ICMP question
>
>
> Hi all,
>
> I have a RedHat 7.3 box which I use as a dual homed firewall.
>
> One of the rules is that I allow all the ICMP traffic from one
> PC in the protected LAN to the Internet.
>
> Here it is one set of rules
>
> iptables -A FORWARD -p ICMP -s 192.168.1.37 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> iptables -A FORWARD -p ICMP -d 192.168.1.37 -m state
> --state ESTABLISHED -j
> ACCEPT
>
> I've been experiencing the problem that with this set of
> rules, suddenly,
> the PC in the LAN cannot "ping" any hosts in the Internet.
> To solve the
> problem
> I delete all my user's chains and flush them, and I worked again.
> I already asked if there is a buffer or counter that gets
> filled, and
> someone
> told me to watch the number of connections being tracked. I
> did that and it
> seems
> that the number of connections being tracked is very low
> (about 160).
>
> Since the problem was a "mistery", I changed the rules to this:
>
> iptables -A FORWARD -p ICMP -s 192.168.1.37 -j ACCEPT
> iptables -A FORWARD -p ICMP -d 192.168.1.37 -j ACCEPT
>
> These are the same rules, except that don't track the state of the
> connections...
>
> Could someone tell me what is the risk asociated to NOT to track the
> connections?
>
> One more question... Is it true that when you specify the
> ICMP protocol,
> iptables only
> applies the rule to the type 0?
>
> thanks in advance
>
> Luis Fernando Barrera
> luba@assist.com.gt
>
>
>
>

Re: ICMP question

[Nick Drage]

On Thu, Nov 14, 2002 at 03:47:18PM -0600, Darrell F. Dieringer wrote:
> 
> Anyone please feel free to correct me if I'm wrong, but I think your
> original rules would work fine if your "ESTABLISHED" rule was changed
> to "ESTABLISHED,RELATED".  Perhaps the "NEW,ESTABLISHED" rule should
> also say "NEW,ESTABLISHED,RELATED" as well.
> 
> If your internal machine issues an "echo request", the "echo reply"
> will be a "RELATED" packet.  (I'd use icmp type numbers, but I don't
> know them off the top of my head.)

/usr/include/netinet/ip_icmp.h

Is always handy for this kind of thing ( 8 and 0 respectively ).

-- 
FunkyJesus System Administration Team

Re: ICMP question

[Joel Newkirk]

On Thursday 14 November 2002 01:28 pm, Luis Fernando Barrera wrote:

> iptables -A FORWARD -p ICMP -s 192.168.1.37 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> iptables -A FORWARD -p ICMP -d 192.168.1.37 -m state --state ESTABLISHED -j
> ACCEPT
>
> I've been experiencing the problem that with this set of rules, suddenly,
> the PC in the LAN cannot "ping" any hosts in the Internet. To solve the
> problem

> iptables -A FORWARD -p ICMP -s 192.168.1.37 -j ACCEPT
> iptables -A FORWARD -p ICMP -d 192.168.1.37 -j ACCEPT
>
> These are the same rules, except that don't track the state of the
> connections...
>
> Could someone tell me what is the risk asociated to NOT to track the
> connections?

For the first rule just use the second version (without state match), but for 
the second rule you want state as it will prevent any ICMP connections 
originating anywhere else from being forwarded to that IP, assuming other 
rules and policy don't ACCEPT it.  (and assuming you don't want that...)  The 
new pair would then allow any ICMP through that came FROM 192.168.1.37, and 
prevent any ICMP from going TO 192.168.1.37 UNLESS it is part of an 
established connection, IE a response to a ping that was ACCEPTed by the 
first rule.

Using the state matches can help you tighten things up considerably, and also 
considerably reduce the number of rules you need to explicitly define.  You 
can allow a local machine out only to port 80 and 53, nothing back to it at 
all, then accept ESTABLISHED/RELATED both ways and everything will work, 
because the responses back to the local machine, and even ICMP messages 
related to the connection, are matched based on state.

> One more question... Is it true that when you specify the ICMP protocol,
> iptables only
> applies the rule to the type 0?

Nope, unless it's a bug in a version other than 1.2.5, I just tested.  (Ping 
my ISP's DNS, nothing, added ICMP rule with no type specified, 18.2ms)

> thanks in advance
>
> Luis Fernando Barrera
> luba@assist.com.gt

j

RE: intermittent and unreliable behaviour with iptables scripts

[alex]

On Wed, 2002-11-13 at 14:34, Doug Watson wrote:
> If there really is a bug in ip_conntrack
> that makes me unfortunately skiddish about 
> continuing on with netfilter/iptables as a 
> viable solution for my company. Yet it seems 
> like many people have implemented this and have
> not seen these types of problems.

Agreed. I finally solved my problem which was due to me not allowing
ICMP packets to sent out the OUTPUT chain for established connections
that where being forwarded. My bad, *not* a conntrack problem, it just
seemed that way at the time.

> I have run the script that you sent me several times.
> Attached is a sample output from it. I don't believe that
> I am seeing anything too strange, but I do have 1 question.
> in the following line which you will see in the attached file
> what does the (policy ACCEPT 4 packets, 284 bytes) mean?
> Chain OUTPUT (policy ACCEPT 4 packets, 284 bytes)

Your attachment seems to of been filtered out. The policy lines mean
that no rules in the chain matched and therefor the default policy took
effect. It is usually recommended to have a default DROP policy if your
being paranoid and explicitly allow the connections you want to come
through. However as I learnt the hard way you have to make sure the
related traffic is also allowed through. The ICMP packet that bit me was
due to fragmentation being required but not possible on the gateway,
probably a symptom of the fact my two interfaces (ppp and eth) had
different MTU's. 

> Is that the total number of packets to traverse the OUTPUT 
> chain or it he number of packets ACCEPTED by the policy for the 
> OUTPUT chain? Or something else?

Just the packets that defaulted to the policy, the other counts will be
against each rule itself.

-- 
Alex
http://www.bennee.com/~alex/