ACK,RST getting dropped in the firewall.

ACK,RST getting dropped in the firewall.

[Manikandan]

Hi Friends,

I am running a stateful firewall in Redhat linux 9 with iptables. I accept
connections, which are ESTABLISHED, RELATED to come inside my LAN and
firewall. I am seeing packets getting dropped which are actually RST packets
of web traffic.  My firewall also blocks incoming connections which are syn
but not new.

My log file is getting filled like this.

Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF PROTO=TCP
SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0
Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF PROTO=TCP
SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0
Jun 23 16:43:22 javagreen kernel: IPT INPUT packet died: IN=eth1 OUT=
MAC=ff:ff:ff:ff:ff:ff:00:0d:60:40:99:db:08:00 SRC=0.0.0.0 DST=255.255.22.255
LEN=340 TOS=0x00 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=320
Jun 23 16:43:26 javagreen kernel: IPT INPUT packet died: IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=4.78.20.2
DST=202.138.22.218 LEN=84 TOS=0x00 PREC=0x00 TTL=41 ID=0 DF PROTO=ICMP
TYPE=8 CODE=0 ID=58217 SEQ=55219
Jun 23 16:43:26 javagreen kernel: IPT INPUT packet died: IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=166.90.213.130
DST=202.138.22.218 LEN=84 TOS=0x00 PREC=0x00 TTL=41 ID=0 DF PROTO=ICMP
TYPE=8 CODE=0 ID=8475 SEQ=60480

Jun 23 16:49:07 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=29723 DF PROTO=TCP
SPT=80 DPT=2193 WINDOW=65073 RES=0x00 ACK URGP=0
Jun 23 16:49:07 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=29723 DF PROTO=TCP
SPT=80 DPT=2193 WINDOW=65073 RES=0x00 ACK URGP=0
Jun 23 16:49:07 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=29748 DF PROTO=TCP
SPT=80 DPT=2194 WINDOW=65063 RES=0x00 ACK URGP=0
Jun 23 16:49:07 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=29748 DF PROTO=TCP
SPT=80 DPT=2194 WINDOW=65063 RES=0x00 ACK URGP=0
Jun 23 16:49:08 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=30132 DF PROTO=TCP
SPT=80 DPT=2192 WINDOW=64507 RES=0x00 ACK URGP=0
Jun 23 16:49:08 javagreen kernel: New not syn:IN=eth0 OUT=
MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=30132 DF PROTO=TCP
SPT=80 DPT=2192 WINDOW=64507 RES=0x00 ACK URGP=0



All the packets come from web sites that have recently been visited and
appear to be the closing down of the TCP connection. It seems that the
netfilter connection tracking is clearing up the connection before it
actually gets closed. netstat on the client machine for the connection shows
the connection in the state CLOSE_WAIT. So far I have only been
seeing these logs for http connections on port 80 and on port 135.


Could someone help me in this?

Thanks in advance.

Regards,
Manikandan.

Re: ACK,RST getting dropped in the firewall.

[Chris Brenton]

On Wed, 2004-06-23 at 07:06, Manikandan wrote:
>
> Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
> MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
> DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF PROTO=TCP
> SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0
> Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
> MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
> DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF PROTO=TCP
> SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0

Seen this a lot. When ever I record a trace it ends up being the
following:
Three packet handshake is normal
Established state goes normally
Client issues a FIN/ACK
State table time-out drops to 2 minutes
Server still has data to send so continues to ACK
State table time-out expires
Server gets blocked at ACK or FIN/ACK stage, session never finishes

There is obviously data getting blocked (based on the packet size) but
I've never had a user complaint. 

> Jun 23 16:43:22 javagreen kernel: IPT INPUT packet died: IN=eth1 OUT=
> MAC=ff:ff:ff:ff:ff:ff:00:0d:60:40:99:db:08:00 SRC=0.0.0.0 DST=255.255.22.255
> LEN=340 TOS=0x00 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=320

You are blocking bootp/DHCP traffic. Should not be a big deal.

> Jun 23 16:43:26 javagreen kernel: IPT INPUT packet died: IN=eth0 OUT=
> MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=4.78.20.2
> DST=202.138.22.218 LEN=84 TOS=0x00 PREC=0x00 TTL=41 ID=0 DF PROTO=ICMP
> TYPE=8 CODE=0 ID=58217 SEQ=55219
> Jun 23 16:43:26 javagreen kernel: IPT INPUT packet died: IN=eth0 OUT=
> MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=166.90.213.130
> DST=202.138.22.218 LEN=84 TOS=0x00 PREC=0x00 TTL=41 ID=0 DF PROTO=ICMP
> TYPE=8 CODE=0 ID=8475 SEQ=60480

You are blocking inbound Ping attempts. Nothing wrong with that. :)

HTH,
Chris

Re: ACK,RST getting dropped in the firewall.

[Gavin Hamill]

On Wednesday 23 June 2004 12:32, Chris Brenton wrote:

> Seen this a lot. When ever I record a trace it ends up being the
> following:

> There is obviously data getting blocked (based on the packet size) but
> I've never had a user complaint.

Just wanted to add a 'me too' - I've asked on-list a couple of times about 
this but never really got an answer. Neither the server nor clients seem to 
be any the worse off for it, but it's obviously incorrect, and it'd be nice 
to be able to nail it =)

Any takers? Antony? :)

gdh

Re: ACK,RST getting dropped in the firewall.

[Antony Stone]

On Thursday 24 June 2004 10:37 am, Gavin Hamill wrote:

> On Wednesday 23 June 2004 12:32, Chris Brenton wrote:
> > Seen this a lot. When ever I record a trace it ends up being the
> > following:
> >
> > There is obviously data getting blocked (based on the packet size) but
> > I've never had a user complaint.
>
> Just wanted to add a 'me too' - I've asked on-list a couple of times about
> this but never really got an answer. Neither the server nor clients seem to
> be any the worse off for it, but it's obviously incorrect, and it'd be nice
> to be able to nail it =)
>
> Any takers? Antony? :)

Hi guys - did somebody call :) ?

Yes, I would agree with the above - blocking RST-ACK does no harm whatever, 
and is often an accidental consequence of using stateful firewalls - they see 
the RST packet (and pass it on), then drop the details from the connection 
tracking table, so by the time the RST-ACK comes along in response, the 
firewall thinks "this isn't part of any established connection, so I'm going 
to drop it" (and maybe log it too if you are logging dropped packets).

Either way, the connection gets dropped as required, and both ends are happy, 
because the end which sent the RST has said bye-bye anyway, and the end which 
recieved it has taken the hint and gone quiet too.

The other thing you see logged quite often for similar reasons is FIN-ACK 
packets - the reply FIN-ACK is regarded as optional by many systems, so 
netfilter often ends up logging (and dropping) the second one to come along.   
No harm done though, because the first one has done the job.

Regards,

Antony.

-- 
Success is a lousy teacher.  It seduces smart people into thinking they can't 
lose.

 - William H Gates III

                                                     Please reply to the list;
                                                           please don't CC me.

Re: ACK,RST getting dropped in the firewall.

[Chris Brenton]

On Thu, 2004-06-24 at 06:44, Antony Stone wrote:
> 
> The other thing you see logged quite often for similar reasons is FIN-ACK 
> packets - the reply FIN-ACK is regarded as optional by many systems, so 
> netfilter often ends up logging (and dropping) the second one to come along.

IMHO this is completely non-RFC. It is clearly defined (RFC 793 page 22)
that both sides must issue a FIN/ACK and receive an ACK before the
connection is closed.

What OS does not issue the reciprocating FIN/ACK? 

> No harm done though, because the first one has done the job.

Actually not quite true as both sides will enter a fin-wait state (RFC
793 page 20) till the second FIN/ACK-ACK exchange takes place. This
chews up a session on both ends till their timers expire. Add up enough
of them and it can kill a busy server (seen it happen and have had to
stop state tracking to fix it).

The "problem" is that there is no requirement for the second FIN to be
immediately issued. The remote host is free to continue to send data
till it is finished. Sometimes this exceeds iptables's timer for the FIN
portion of the session, thus causing the problem many of us are seeing. 

Also, if you check the log entry that was posted:

> Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
> MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
> DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF 
> PROTO=TCP SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0

Its a 1500 byte ACK packet. So iptables is not just blocking the FIN/ACK
exchange, it can sometimes end up blocking data transfer as well. :(

I reported this "bug" back when iptables was still alpha code. At the
time iptables would expire the session 60 seconds after the first FIN
has been passed. I think Rusty bumped the timer up to 2 minutes. Perhaps
its been tweaked back down again or needs to be increased to 3 minutes?

HTH,
Chris

Re: ACK,RST getting dropped in the firewall.

[Jozsef Kadlecsik]

On Thu, 24 Jun 2004, Chris Brenton wrote:

> On Thu, 2004-06-24 at 06:44, Antony Stone wrote:
> >
> > The other thing you see logged quite often for similar reasons is FIN-ACK
> > packets - the reply FIN-ACK is regarded as optional by many systems, so
> > netfilter often ends up logging (and dropping) the second one to come along.
>
> IMHO this is completely non-RFC. It is clearly defined (RFC 793 page 22)
> that both sides must issue a FIN/ACK and receive an ACK before the
> connection is closed.
>
> What OS does not issue the reciprocating FIN/ACK?
>
> > No harm done though, because the first one has done the job.
>
> Actually not quite true as both sides will enter a fin-wait state (RFC
> 793 page 20) till the second FIN/ACK-ACK exchange takes place. This
> chews up a session on both ends till their timers expire. Add up enough
> of them and it can kill a busy server (seen it happen and have had to
> stop state tracking to fix it).
>
> The "problem" is that there is no requirement for the second FIN to be
> immediately issued. The remote host is free to continue to send data
> till it is finished. Sometimes this exceeds iptables's timer for the FIN
> portion of the session, thus causing the problem many of us are seeing.
>
> Also, if you check the log entry that was posted:
>
> > Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
> > MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
> > DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF
> > PROTO=TCP SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0
>
> Its a 1500 byte ACK packet. So iptables is not just blocking the FIN/ACK
> exchange, it can sometimes end up blocking data transfer as well. :(

That should not happen, except when the host does not respond after
receiving the FIN/ACK by more than 2 minutes.

> I reported this "bug" back when iptables was still alpha code. At the
> time iptables would expire the session 60 seconds after the first FIN
> has been passed. I think Rusty bumped the timer up to 2 minutes. Perhaps
> its been tweaked back down again or needs to be increased to 3 minutes?

Could you try the TCP window tracking patch from pom-ng which improves TCP
connection tracking a lot? The original code cannot really be improved
besides tuning the timeout parameters.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ACK,RST getting dropped in the firewall.

[Antony Stone]

On Thursday 24 June 2004 2:36 pm, Chris Brenton wrote:

> On Thu, 2004-06-24 at 06:44, Antony Stone wrote:
> > The other thing you see logged quite often for similar reasons is FIN-ACK
> > packets - the reply FIN-ACK is regarded as optional by many systems, so
> > netfilter often ends up logging (and dropping) the second one to come
> > along.
>
> IMHO this is completely non-RFC. It is clearly defined (RFC 793 page 22)
> that both sides must issue a FIN/ACK and receive an ACK before the
> connection is closed.

Oh, I agree it's RFC incompliant.   I didn't say it was official behaviour - 
just something you see happening :(

> What OS does not issue the reciprocating FIN/ACK?

Don't know off the top of my head - will let you know if I can find out.

> > No harm done though, because the first one has done the job.
>
> Actually not quite true as both sides will enter a fin-wait state (RFC
> 793 page 20) till the second FIN/ACK-ACK exchange takes place.

Depends what you mean by "done the job".   I meant the phrase in the sense of 
"no more data is going to pass across the connection".

> This chews up a session on both ends till their timers expire. Add up enough
> of them and it can kill a busy server (seen it happen and have had to
> stop state tracking to fix it).

Indeed.   Not necessarily as bad as a SYN flood attack, but similar in nature 
and consequences.

> Also, if you check the log entry that was posted:
> > Jun 23 16:42:43 javagreen kernel: New not syn:IN=eth0 OUT=
> > MAC=00:09:6b:19:b4:24:00:0e:83:f6:19:9f:08:00 SRC=202.138.101.5
> > DST=202.138.22.218 LEN=1500 TOS=0x00 PREC=0x00 TTL=122 ID=51601 DF
> > PROTO=TCP SPT=80 DPT=2162 WINDOW=64574 RES=0x00 ACK URGP=0
>
> Its a 1500 byte ACK packet. So iptables is not just blocking the FIN/ACK
> exchange, it can sometimes end up blocking data transfer as well. :(

Oh - I didn't see that posting - I was replying to a question apparently about 
RST-ACK packets getting logged (and dropped).

Dropping a normal ACK packet is another matter altogether, and clearly a Bad 
Thing (or at least, Highly Undesirable).

Regards,

Antony.

-- 
"Reports that say that something hasn't happened are always interesting to me, 
because as we know, there are known knowns; there are things we know we know. 
We also know there are known unknowns; that is to say we know there are some 
things we do not know. But there are also unknown unknowns - the ones we 
don't know we don't know."

 - Donald Rumsfeld, US Secretary of Defence

                                                     Please reply to the list;
                                                           please don't CC me.

Re: ACK,RST getting dropped in the firewall.

[Chris Brenton]

On Thu, 2004-06-24 at 09:54, Antony Stone wrote:
>
> > > No harm done though, because the first one has done the job.
> >
> > Actually not quite true as both sides will enter a fin-wait state (RFC
> > 793 page 20) till the second FIN/ACK-ACK exchange takes place.
> 
> Depends what you mean by "done the job".   I meant the phrase in the sense of 
> "no more data is going to pass across the connection".

I *think* this may be the point of confusion. A FIN/ACK only means that
the _sender_ is done transmitting data, To quote from 793:

"The user who CLOSEs may continue to RECEIVE until he is told that the
other side has CLOSED also.  Thus, a program could initiate several
SENDs followed by a CLOSE, and then continue to RECEIVE until signaled
that a RECEIVE failed because the other side has CLOSED."

So just because one side of the connection issues a FIN/ACK, it does not
mean the session is over. The other side of the session may have a lot
more data to transmit so the session can stay active for some additional
period of time. Here in lies out problem. Netfilter is reacting to the
first FIN/ACK and killing off the session before the other end finishes
communicating and issues it's own FIN/ACK. 

The problem *appears to be* (based on my limited testing) in how the
timer is implemented. In an ESTABLISHED state there is a 5 day timer
that gets reset when ever a new packet crosses the firewall. The FIN/ACK
timer seems to be an absolute timer that kicks off at the first FIN/ACK
and continues to count down regardless of whether the session is still
active or not. If it acted more like the ESTABLISHED timer, this problem
would go away.

I can understand wanting a low timer to conserve resources. At the same
time however its killing legitimate RFC compliant sessions. :(

HTH,
Chris

Re: ACK,RST getting dropped in the firewall.

[Antony Stone]

On Thursday 24 June 2004 4:48 pm, Chris Brenton wrote:

> On Thu, 2004-06-24 at 09:54, Antony Stone wrote:
> > > > No harm done though, because the first one has done the job.
> > >
> > > Actually not quite true as both sides will enter a fin-wait state (RFC
> > > 793 page 20) till the second FIN/ACK-ACK exchange takes place.
> >
> > Depends what you mean by "done the job".   I meant the phrase in the
> > sense of "no more data is going to pass across the connection".
>
> I *think* this may be the point of confusion. A FIN/ACK only means that
> the _sender_ is done transmitting data.

Agreed.

> So just because one side of the connection issues a FIN/ACK, it does not
> mean the session is over. The other side of the session may have a lot
> more data to transmit so the session can stay active for some additional
> period of time. Here in lies out problem. Netfilter is reacting to the
> first FIN/ACK and killing off the session before the other end finishes
> communicating and issues it's own FIN/ACK.

In my experience (with things like SMTP, HTTP) it appears to be the server 
which generally issues the (first) FIN/ACK, telling the client that there's 
no more data (by which time the client has finished anyway, since it issued 
the request and was just waiting for the response to come back).   It seems 
not to be often that the client sends a FIN/ACK once it's finished sending 
the request, knowing that there's plenty of data to come in a reply (if there 
were, I think netfilter would have been changed quite some time ago).

I wonder how other stateful firewalls (eg: CP FW-1?) handle this sort of thing 
- do they listen for *both* FIN/ACKs (and thereby leave themselves prey to 
half-open connections littering their connection table for systems which 
don't bother to send the responding FIN/ACK)?

Regards,

Antony.

-- 
There's no such thing as bad weather - only the wrong clothes.

 - Billy Connolly

                                                     Please reply to the list;
                                                           please don't CC me.

Re: ACK,RST getting dropped in the firewall.

[Chris Brenton]

On Thu, 2004-06-24 at 12:09, Antony Stone wrote:
>
> In my experience (with things like SMTP, HTTP) it appears to be the server 
> which generally issues the (first) FIN/ACK, telling the client that there's 
> no more data 

Weird, my experience with HTTP has been the other way around. The client
request typically requires just a single back. Its the data returned
from the server that can take multiple packets (thus the client finishes
first). This is why when most people run into this time-out issue its
always the server still trying to transmit _from_ TCP/80, rather than
the client trying to send _to_ TCP/80. A good example are the log
entries that started this thread.

> I wonder how other stateful firewalls (eg: CP FW-1?) handle this sort of thing 
> - do they listen for *both* FIN/ACKs (and thereby leave themselves prey to 
> half-open connections littering their connection table for systems which 
> don't bother to send the responding FIN/ACK)?

With FW-1 and PIX, they use a higher state table time out. Again, proper
way to fix this would be to reset the timer if the session stays active.
Fixes both your worries and mine. :)

As for OS's that don't return a FIN/ACK, I can't say I've seen this in
the wild (which is why I asked which OS so I could test this). I know
there used to be a few OS's (older Mac OS and SGI come to mind) that
used to "cheat" by sending a RST/ACK instead of a FIN/ACK, but that was
cleaned up years ago.

HTH,
Chris