[BUG] SNAT sometimes allows packets to pass through unchanged

[BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

I'm seeing some strange behavior on my firewall, which is running 
Fedora 8's version of 2.6.26.  Every so often a packet with a private 
source address is sent out the public interface unchanged, when it 
should be dropped.

This happens when internal hosts are slow to close their end of a TCP 
connection.  For example:

	Internal host A (using a private address) initiates a TCP
	connection to an external server B.

	Data is sent back and forth.

	External host B sends a FIN and host A responds with ACK.

	Several minutes later (after the tracking for this connection
	has expired), host A sends a FIN to host B.  This packet
	goes through the firewall unchanged and is sent out the
	public interface with the private source address intact.

Now I would expect that such packets would be dropped, because they 
don't belong to an existing connection and they can't be the start of a 
new connection.  The fact that this doesn't happen indicates there is a 
bug in the netfilter code somewhere.

For reference, here is a lightly-edited extract from my iptables 
script on the firewall:

------------------------------------------------------------------------
# Use SNAT for new outgoing connections from private addresses.
OUTIP=...  # The IP address of the firewall's public interface, eth1
iptables -t nat -N startnat
iptables -t nat -A startnat -p tcp -j SNAT --to-source $OUTIP:7000-65500
iptables -t nat -A startnat -p udp -j SNAT --to-source $OUTIP:7000-65500
iptables -t nat -A startnat -p icmp -j SNAT --to-source $OUTIP
iptables -t nat -A startnat -j DROP

iptables -t nat -A POSTROUTING -o eth1 -s 10.0.0.0/8 -j startnat
------------------------------------------------------------------------

Can anyone help figure out what's wrong?

Alan Stern

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Patrick McHardy]

Alan Stern wrote:
> I'm seeing some strange behavior on my firewall, which is running 
> Fedora 8's version of 2.6.26.  Every so often a packet with a private 
> source address is sent out the public interface unchanged, when it 
> should be dropped.
> 
> This happens when internal hosts are slow to close their end of a TCP 
> connection.  For example:
> 
> 	Internal host A (using a private address) initiates a TCP
> 	connection to an external server B.
> 
> 	Data is sent back and forth.
> 
> 	External host B sends a FIN and host A responds with ACK.
> 
> 	Several minutes later (after the tracking for this connection
> 	has expired), host A sends a FIN to host B.  This packet
> 	goes through the firewall unchanged and is sent out the
> 	public interface with the private source address intact.
> 
> Now I would expect that such packets would be dropped, because they 
> don't belong to an existing connection and they can't be the start of a 
> new connection.  The fact that this doesn't happen indicates there is a 
> bug in the netfilter code somewhere.

If the connection has already timed out (from conntracks perspective),
it has lost its state. Unless connection pickup is enabled, the packet 
will be marked as INVALID because it doesn't belong to a connection.
You can control dropping of these packets yourself by adding the
appropriate "-m state --state INVALID" rules. That said, there were
some bugs in the past few releases that caused some bad interaction
between TCP and TCP conntrack (not sure anymore which one of both was
to blame). Its possible that this is the root cause for this, so
you might want to consider a kernel update.

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

On Thu, 12 Feb 2009, Patrick McHardy wrote:

> If the connection has already timed out (from conntracks perspective),
> it has lost its state. Unless connection pickup is enabled, the packet 
> will be marked as INVALID because it doesn't belong to a connection.
> You can control dropping of these packets yourself by adding the
> appropriate "-m state --state INVALID" rules.

I tried adding a rule to log these unaccounted-for packets.  Nothing 
showed up, even when I could see the packets being sent.

> That said, there were
> some bugs in the past few releases that caused some bad interaction
> between TCP and TCP conntrack (not sure anymore which one of both was
> to blame). Its possible that this is the root cause for this, so
> you might want to consider a kernel update.

It does sound like the result of a bug.  Do you have any pointers to 
patches or locations to check in the source?

Alan Stern

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Patrick McHardy]

Alan Stern wrote:
> On Thu, 12 Feb 2009, Patrick McHardy wrote:
> 
>> If the connection has already timed out (from conntracks perspective),
>> it has lost its state. Unless connection pickup is enabled, the packet 
>> will be marked as INVALID because it doesn't belong to a connection.
>> You can control dropping of these packets yourself by adding the
>> appropriate "-m state --state INVALID" rules.
> 
> I tried adding a rule to log these unaccounted-for packets.  Nothing 
> showed up, even when I could see the packets being sent.

Where (table/chain/position) did you add this rule?

>> That said, there were
>> some bugs in the past few releases that caused some bad interaction
>> between TCP and TCP conntrack (not sure anymore which one of both was
>> to blame). Its possible that this is the root cause for this, so
>> you might want to consider a kernel update.
> 
> It does sound like the result of a bug.  Do you have any pointers to 
> patches or locations to check in the source?

Sorry, there were quite a few patches and I don't remember which
ones exactly are related.

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

On Mon, 16 Feb 2009, Patrick McHardy wrote:

> Alan Stern wrote:
> > On Thu, 12 Feb 2009, Patrick McHardy wrote:
> > 
> >> If the connection has already timed out (from conntracks perspective),
> >> it has lost its state. Unless connection pickup is enabled, the packet 
> >> will be marked as INVALID because it doesn't belong to a connection.
> >> You can control dropping of these packets yourself by adding the
> >> appropriate "-m state --state INVALID" rules.
> > 
> > I tried adding a rule to log these unaccounted-for packets.  Nothing 
> > showed up, even when I could see the packets being sent.
> 
> Where (table/chain/position) did you add this rule?

In the first position of the POSTROUTING chain in the nat table.  I
don't remember exactly what rules I used, but at one point I tried
something very much like this:

iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn

The counter for this rule remained at 0 even after packets with private 
source addresses were sent through the public interface.

> >> That said, there were
> >> some bugs in the past few releases that caused some bad interaction
> >> between TCP and TCP conntrack (not sure anymore which one of both was
> >> to blame). Its possible that this is the root cause for this, so
> >> you might want to consider a kernel update.
> > 
> > It does sound like the result of a bug.  Do you have any pointers to 
> > patches or locations to check in the source?
> 
> Sorry, there were quite a few patches and I don't remember which
> ones exactly are related.

I tried using 2 6.27 kernel but the problem remained.  Building a later 
version won't be easy because of the need to create the proper config.  
Can you remember in which version these bugs got fixed?

Alan Stern

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Patrick McHardy]

Alan Stern wrote:
> On Mon, 16 Feb 2009, Patrick McHardy wrote:
> 
>>> I tried adding a rule to log these unaccounted-for packets.  Nothing 
>>> showed up, even when I could see the packets being sent.
>> Where (table/chain/position) did you add this rule?
> 
> In the first position of the POSTROUTING chain in the nat table.  I
> don't remember exactly what rules I used, but at one point I tried
> something very much like this:
> 
> iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn
> 
> The counter for this rule remained at 0 even after packets with private 
> source addresses were sent through the public interface.

The NAT table only sees the first packet of every connection
and never INVALID packets. The mangle table should work fine.

You can also enable conntrack-internal logging of invalid packets:

echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

>> Sorry, there were quite a few patches and I don't remember which
>> ones exactly are related.
> 
> I tried using 2 6.27 kernel but the problem remained.  Building a later 
> version won't be easy because of the need to create the proper config.  
> Can you remember in which version these bugs got fixed?

Sorry, no.

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

On Mon, 16 Feb 2009, Patrick McHardy wrote:

> Alan Stern wrote:
> > On Mon, 16 Feb 2009, Patrick McHardy wrote:
> > 
> >>> I tried adding a rule to log these unaccounted-for packets.  Nothing 
> >>> showed up, even when I could see the packets being sent.
> >> Where (table/chain/position) did you add this rule?
> > 
> > In the first position of the POSTROUTING chain in the nat table.  I
> > don't remember exactly what rules I used, but at one point I tried
> > something very much like this:
> > 
> > iptables -t nat -I POSTROUTING 1 -s 10.0.0.0/8 -p tcp ! --syn
> > 
> > The counter for this rule remained at 0 even after packets with private 
> > source addresses were sent through the public interface.
> 
> The NAT table only sees the first packet of every connection
> and never INVALID packets. The mangle table should work fine.

Okay, I'll try it instead.  Are these facts documented anywhere?  FWIW, 
I don't recall ever seeing anywhere a description of what packets go 
through the mangle table, or at what stage of processing.

> You can also enable conntrack-internal logging of invalid packets:
> 
> echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

I did this (except I used the value of IPPROTO_TCP instead of 255).  
Several things did get logged, but no messages were produced for the
packets I'm concerned about.  _That_ definitely seemed like a bug...

> > I tried using 2 6.27 kernel but the problem remained.  Building a later 
> > version won't be easy because of the need to create the proper config.  
> > Can you remember in which version these bugs got fixed?
> 
> Sorry, no.

Hm.  Maybe I'll try to set up a test system with a 2.6.29-rc kernel, to 
see if the problem really has been fixed yet.

Alan Stern

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

On Mon, 16 Feb 2009, Patrick McHardy wrote:

> The NAT table only sees the first packet of every connection
> and never INVALID packets. The mangle table should work fine.

I ended up adding a rule to the FORWARD chain of the filter table.  
The trick was to select based on the state.  That worked; it saw all
those un-NATed packets and was able to eliminate them.  In case you're
curious, the rule was essentially this:

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP

Ideally, the rule should select all the packets which haven't been
altered by SNAT, not just the ones marked INVALID.  Is there any way to
do this?

Alan Stern

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Patrick McHardy]

Alan Stern wrote:
> On Mon, 16 Feb 2009, Patrick McHardy wrote:
> 
>> The NAT table only sees the first packet of every connection
>> and never INVALID packets. The mangle table should work fine.
> 
> I ended up adding a rule to the FORWARD chain of the filter table.  
> The trick was to select based on the state.  That worked; it saw all
> those un-NATed packets and was able to eliminate them.  In case you're
> curious, the rule was essentially this:
> 
> iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP
 >
> Ideally, the rule should select all the packets which haven't been
> altered by SNAT, not just the ones marked INVALID.  Is there any way to
> do this?

Not in the sense that you could somehow catch valid packets "missed"
by SNAT, that would be a bug. The conntrack match supports matching
on whether the state says that a packet should be NATed.

Re: [BUG] SNAT sometimes allows packets to pass through unchanged

[Alan Stern]

On Wed, 18 Feb 2009, Patrick McHardy wrote:

> Alan Stern wrote:
> > On Mon, 16 Feb 2009, Patrick McHardy wrote:
> > 
> >> The NAT table only sees the first packet of every connection
> >> and never INVALID packets. The mangle table should work fine.
> > 
> > I ended up adding a rule to the FORWARD chain of the filter table.  
> > The trick was to select based on the state.  That worked; it saw all
> > those un-NATed packets and was able to eliminate them.  In case you're
> > curious, the rule was essentially this:
> > 
> > iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m state --state INVALID -j DROP
>  >
> > Ideally, the rule should select all the packets which haven't been
> > altered by SNAT, not just the ones marked INVALID.  Is there any way to
> > do this?
> 
> Not in the sense that you could somehow catch valid packets "missed"
> by SNAT, that would be a bug. The conntrack match supports matching
> on whether the state says that a packet should be NATed.

You mean I should do something like this?

iptables -A FORWARD -o eth1 -s 10.0.0.0/8 -m conntrack --ctstate ! SNAT -j DROP

The man page says that --ctstate doesn't support using ! for negation,
but that detail is easily worked around.

Alan Stern