SNAT and FIN/ACK packets after a long time

SNAT and FIN/ACK packets after a long time

[Carsten Aulbert]

Hi all,

we have a weird problem here, which we don't know if we understand it 
correctly:

We have one "router" which uses netfilter/iptables to do SNAT and this works 
in principle. However, we have a program that seems to have a bug and leaves a 
connection open in CLOSE_WAIT state for (probably) a very long time (hours, 
days). Thus, the program probably receives a FIN but does not ACK that.

Now, when we kill this program, it seems to realize it's error and sends the 
FIN/ACK packet out. 

Here we get a nice netfilter problem. It seems the packets arrive at the 
internal iface of the SNAT machine, go through the FORWARD chain, but then 
bypass the POSTROUTING chain, as we see the internally used 10.0.0.0/8 
addresses on the external iface going out into the world.

Has anyone seen that?

here's our SNAT from iptables-save:

# Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
*mangle
:PREROUTING ACCEPT [33564248:152410100983]
:INPUT ACCEPT [30142982:149275397517]
:FORWARD ACCEPT [3415403:3134487334]
:OUTPUT ACCEPT [22834014:4427390138]
:POSTROUTING ACCEPT [26247062:7561729612]
COMMIT
# Completed on Wed Jan 20 15:12:02 2010
# Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
*nat
:PREROUTING ACCEPT [426653254:210106871813]
:POSTROUTING ACCEPT [86678890:27123554142]
:OUTPUT ACCEPT [88632375:27265224806]
-A POSTROUTING -o eth2 -j SNAT --to-source a.b.c.d
-A POSTROUTING -s 10.0.0.0/8 -o eth2 -j LOG --log-prefix "BAD POSTROUTING: "
COMMIT
# Completed on Wed Jan 20 15:12:02 2010
# Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
*filter
:INPUT DROP [682:213218]
:FORWARD DROP [2918:175080]
:OUTPUT ACCEPT [58195797:29502612523]
:fail2ban-ssh - [0:0]
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth3 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -m state --state 
NEW,RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 40000:45000 -j ACCEPT
-A INPUT -i eth2 -p tcp -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i eth0 -o eth2 -j ACCEPT
-A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -s 10.0.0.0/8 -o eth2 -j LOG --log-prefix "BAD OUTPUT: "
-A OUTPUT -s 10.0.0.0/8 -o eth2 -j DROP
COMMIT
# Completed on Wed Jan 20 15:12:02 2010

Is it possible that netfilter's connection tracker looses track of the old 
connections but does not drop this packet?

Thanks for any hint

Carsten

Re: SNAT and FIN/ACK packets after a long time

[Mart Frauenlob]

On 20.01.2010 15:13, Carsten Aulbert wrote:
> Hi all,
> 
> we have a weird problem here, which we don't know if we understand it 
> correctly:
> 
> We have one "router" which uses netfilter/iptables to do SNAT and this works 
> in principle. However, we have a program that seems to have a bug and leaves a 
> connection open in CLOSE_WAIT state for (probably) a very long time (hours, 
> days). Thus, the program probably receives a FIN but does not ACK that.
> 
> Now, when we kill this program, it seems to realize it's error and sends the 
> FIN/ACK packet out. 
> 
> Here we get a nice netfilter problem. It seems the packets arrive at the 
> internal iface of the SNAT machine, go through the FORWARD chain, but then 
> bypass the POSTROUTING chain, as we see the internally used 10.0.0.0/8 
> addresses on the external iface going out into the world.
> 
> Has anyone seen that?
> 
> here's our SNAT from iptables-save:
> 
> # Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
> *mangle
> :PREROUTING ACCEPT [33564248:152410100983]
> :INPUT ACCEPT [30142982:149275397517]
> :FORWARD ACCEPT [3415403:3134487334]
> :OUTPUT ACCEPT [22834014:4427390138]
> :POSTROUTING ACCEPT [26247062:7561729612]
> COMMIT
> # Completed on Wed Jan 20 15:12:02 2010
> # Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
> *nat
> :PREROUTING ACCEPT [426653254:210106871813]
> :POSTROUTING ACCEPT [86678890:27123554142]
> :OUTPUT ACCEPT [88632375:27265224806]
> -A POSTROUTING -o eth2 -j SNAT --to-source a.b.c.d
> -A POSTROUTING -s 10.0.0.0/8 -o eth2 -j LOG --log-prefix "BAD POSTROUTING: "
> COMMIT
> # Completed on Wed Jan 20 15:12:02 2010
> # Generated by iptables-save v1.4.2 on Wed Jan 20 15:12:02 2010
> *filter
> :INPUT DROP [682:213218]
> :FORWARD DROP [2918:175080]
> :OUTPUT ACCEPT [58195797:29502612523]
> :fail2ban-ssh - [0:0]
> -A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -i lo -j ACCEPT
> -A INPUT -i eth0 -j ACCEPT
> -A INPUT -i eth3 -j ACCEPT
> -A INPUT -p icmp -m icmp --icmp-type 8 -m state --state 
> NEW,RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -i eth2 -p tcp -m tcp --dport 22 -j ACCEPT
> -A INPUT -i eth2 -p tcp -m tcp --dport 40000:45000 -j ACCEPT
> -A INPUT -i eth2 -p tcp -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i eth0 -o eth2 -j ACCEPT
> -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A OUTPUT -s 10.0.0.0/8 -o eth2 -j LOG --log-prefix "BAD OUTPUT: "
> -A OUTPUT -s 10.0.0.0/8 -o eth2 -j DROP
> COMMIT
> # Completed on Wed Jan 20 15:12:02 2010
> 
> Is it possible that netfilter's connection tracker looses track of the old 
> connections but does not drop this packet?
> 
> Thanks for any hint
> 
> Carsten

The connection tracking will most likely recognize those packet as in
'INVALID' state. You do not drop them (in FORWARD chain).
The nat table does not see packets in 'INVALID' state (needs
'normalized' connections).

-t filter ... -m state --state INVALID ... -j DROP

at a fitting place with additional matches at will.

best regards

Mart

Re: SNAT and FIN/ACK packets after a long time

[Carsten Aulbert]

On Wednesday 20 January 2010 16:23:53 Mart Frauenlob wrote:

> -t filter ... -m state --state INVALID ... -j DROP
> 

That's it, you nailed that one.

Sorry, that we missed it, maybe we have just too many well written/behaving 
programs ;)

Cheers

Carsten