2.4 SNAT fails randomly

2.4 SNAT fails randomly

[Tim Gardner]

I think I have a case where SNAT is sometimes failing. I have a specific case 
where UDP SNAT fails repeatedly. However, I have SPOOF log entries on some of 
our servers that indicate TCP is also failing SNAT once in awhile.

In particular, I (as an ISP) have a client that repeatably demonstrates the 
problem. This client is behind our embedded Linux 2.4.19 StrongARM NAT 
firewall using P-O-M from CVS as of 11/14/2003. I have applied the following 
P-O-M patches:

submitted/02_2.4.20.patch
submitted/03_2.4.21.patch
submitted/04_2.4.22.patch
base/mport.patch"
base/iprange.patch"
base/psd.patch"
base/connlimit.patch"
base/u32.patch"
extra/h323-conntrack-nat.patch"
extra/pptp-conntrack-nat.patch"

This client has a NetGear router connected to our firewall. The NetGear router 
has a well known bug wherein it attempts to get time/date via UDP/NTP from a 
hardcoded destination (caesar.cs.wisc.edu). This is the UDP packet that is 
not being SNAT'd. The packet in question is thus:

tcpdump  -vvv -t -i eth1 -p -x -X -e 'port 123'
tcpdump: listening on eth1
0:2:6f:3:60:5c 0:90:27:90:49:e7 ip 90: 192.168.10.2.23457 > 
caesar.cs.wisc.edu.ntp:  v1 client strat 0 poll 0 prec 0 dist 0.000000 disp 
0.000000 ref (unspec)@0.000000000 [|ntp] (ttl 253, id 63424, len 76)
0x0000   4500 004c f7c0 0000 fd11 53c1 c0a8 0a02        E..L......S.....
0x0010   8069 270b 5ba1 007b 0038 2643 0b00 0000        .i'.[..{.8&C....
0x0020   0000 0000 0000 0000 0000 0000 0000 0000        ................
0x0030   0000 0000 0000

I have verfied that this really is the packet that is being emitted on eth1 
and is not an artifact of tcpdump. As you can see from the Chain POSTROUTING 
output below, some packets are clearly getting SNAT'd. Why not all of them?

The iptables rules are straightforward. 
---------------------------------------

IPTABLES=${IPTABLES:-/bin/iptables}
EXTIF=eth1
INTIF=eth0
${IPTABLES} -P INPUT ACCEPT
${IPTABLES} -F INPUT
${IPTABLES} -P OUTPUT ACCEPT
${IPTABLES} -F OUTPUT
${IPTABLES} -P FORWARD DROP
${IPTABLES} -F FORWARD
${IPTABLES} -t nat -F
#
# Established or NAT helper streams get out of jail free.
#
${IPTABLES} -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
#
# Block client hosting of well known services as well as SMB noise.
#
${IPTABLES} -A FORWARD -s 192.168.10.2/32 -p tcp -m mport --sports 
21,23,25,80,110,137:139 -j DROP
${IPTABLES} -A FORWARD -s 192.168.10.2/32 -p udp -m mport --sports 
137:139,201:206,396 -j DROP
${IPTABLES} -A FORWARD -d 192.168.10.2/32 -p tcp -m mport --dports 
137:139,201:206,396 -j DROP
${IPTABLES} -A FORWARD -d 192.168.10.2/32 -p udp -m mport --dports 
137:139,201:206,396 -j DROP
#
# Estable SNAT/DNAT mapping.
#
${IPTABLES} -t nat -A POSTROUTING -s 192.168.10.2 -j SNAT --to-source 
216.166.171.155
${IPTABLES} -t nat -A PREROUTING -d 216.166.171.155 -j DNAT --to 192.168.10.2
#
# Allow ARP broadcasts from client.
#
${IPTABLES} -A FORWARD -s 192.168.10.0/24 -d 255.255.255.255 -j ACCEPT
#
# Block multicast from client.
#
${IPTABLES} -A FORWARD -s 192.168.10.0/24 -d 224.0.0.0/3 -j DROP
#
# Allow icmp echo-request to client, but block pings from client.
#
${IPTABLES} -A FORWARD -d 192.168.10.0/24 -p icmp --icmp-type echo-request -j 
ACCEPT
${IPTABLES} -A FORWARD -s 192.168.10.0/24 -p icmp --icmp-type echo-request -j 
DROP
#
# Allow inbound non-TCP traffic, e.g., IPSEC and UDP.
#
${IPTABLES} -A FORWARD -i ${EXTIF} -o ${INTIF} ! -p tcp -m iprange --dst-range 
192.168.10.2-192.168.10.2 -j ACCEPT
#
# Allow inbound new TCP connections.
#
${IPTABLES} -A FORWARD -i ${EXTIF} -o ${INTIF} -d 192.168.10.0/24 -m state 
--state NEW -j ACCEPT
#
# Allow client outbound packets from known source addresses. These packets 
should all go to
# the POSTROUTING rule.
# 
${IPTABLES} -A FORWARD -i ${INTIF} -o ${EXTIF} -m iprange --src-range 
192.168.10.2-192.168.10.2 -j ACCEPT

This is what the iptables dump shows:
-------------------------------------

Chain INPUT (policy ACCEPT 786 packets, 67921 bytes)
 pkts bytes target     prot opt in     out     source               
destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               
destination
   31  2608 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
state RELATED,ESTABLISHED
    0     0 DROP       tcp  --  *      *       192.168.10.2         0.0.0.0/0           
mport sports 21,23,25,80,110,137:139
    0     0 DROP       udp  --  *      *       192.168.10.2         0.0.0.0/0           
mport sports 137:139,201:206,396
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            
192.168.10.2        mport dports 137:139,201:206,396
    0     0 DROP       udp  --  *      *       0.0.0.0/0            
192.168.10.2        mport dports 137:139,201:206,396
    0     0 ACCEPT     all  --  *      *       192.168.10.0/24      
255.255.255.255
    0     0 DROP       all  --  *      *       192.168.10.0/24      
224.0.0.0/3
    1    92 ACCEPT     icmp --  *      *       0.0.0.0/0            
192.168.10.0/24     icmp type 8
    0     0 DROP       icmp --  *      *       192.168.10.0/24      0.0.0.0/0           
icmp type 8
    1   404 ACCEPT    !tcp  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0           
destination IP range 192.168.10.2-192.168.10.2
    2   120 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            
192.168.10.0/24     state NEW
 2348  179K ACCEPT     all  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0           
source IP range 192.168.10.2-192.168.10.2

Chain OUTPUT (policy ACCEPT 527 packets, 74202 bytes)
 pkts bytes target     prot opt in     out     source               
destination

Chain PREROUTING (policy ACCEPT 811 packets, 71641 bytes)
 pkts bytes target     prot opt in     out     source               
destination
    3   556 DNAT       all  --  *      *       0.0.0.0/0            
216.166.171.155     to:192.168.10.2

Chain POSTROUTING (policy ACCEPT 53 packets, 4657 bytes)
 pkts bytes target     prot opt in     out     source               
destination
    6   572 SNAT       all  --  *      *       192.168.10.2         0.0.0.0/0           
to:216.166.171.155

Chain OUTPUT (policy ACCEPT 49 packets, 4025 bytes)
 pkts bytes target     prot opt in     out     source               
destination


Here are my interface definitions:
----------------------------------

eth0      Link encap:Ethernet  HWaddr 00:D0:CA:F1:20:A4  
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:414 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:31413 (30.6 kb)  TX bytes:228 (228.0 b)
          Interrupt:47 Base address:0x300 

eth0:10   Link encap:Ethernet  HWaddr 00:D0:CA:F1:20:A4  
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          Interrupt:47 Base address:0x300 

eth1      Link encap:Ethernet  HWaddr 00:02:6F:03:60:5C  
          inet addr:172.16.233.252  Bcast:172.16.239.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4072 errors:0 dropped:0 overruns:0 frame:0
          TX packets:645 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:310754 (303.4 kb)  TX bytes:86412 (84.3 kb)

eth1:10-2 Link encap:Ethernet  HWaddr 00:02:6F:03:60:5C  
          inet addr:216.166.171.155  Bcast:216.166.171.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

rf0       Link encap:Ethernet  HWaddr 00:02:6F:03:60:5C  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16964 errors:0 dropped:0 overruns:0 frame:0
          TX packets:691 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:765858 (747.9 kb)  TX bytes:71316 (69.6 kb)
          Interrupt:43 


-- 
Tim Gardner - timg@tpi.com 406-443-5357
TriplePoint, Inc. - http://www.tpi.com
PGP: http://www.tpi.com/PGP/Tim.txt

Re: 2.4 SNAT fails randomly

[Balazs Scheidler]

On Sat, Nov 15, 2003 at 11:03:47AM -0700, Tim Gardner wrote:
> I think I have a case where SNAT is sometimes failing. I have a specific case 
> where UDP SNAT fails repeatedly. However, I have SPOOF log entries on some of 
> our servers that indicate TCP is also failing SNAT once in awhile.
>[snip] 
> This client has a NetGear router connected to our firewall. The NetGear router 
> has a well known bug wherein it attempts to get time/date via UDP/NTP from a 
> hardcoded destination (caesar.cs.wisc.edu). This is the UDP packet that is 
> not being SNAT'd. The packet in question is thus:
> 

NAT mappings are established for NEW connections only, isn't it possible that your 
client sent an NTP request while your rule was not yet established? This means that it
is entered the CONNTRACK table without the NAT manip and anything that comes later is
not NATed as it is not a fresh CONNTRACK.

Try filtering this traffic for 180secs and see it disappear from /proc/net/ip_conntrack, 
then remove the filtering and check whether it traversed the  nat/POSTROUTING chain.

-- 
Bazsi

Re: 2.4 SNAT fails randomly

[Tim Gardner]

Balazs,

You are exactly correct. I swapped the interface creation statements with the 
iptables rules so that the rules were in place before any packets could be 
received from the interior net. In fact, I had to make sure that the interior 
net interface was down. Otherwise it would make an entry in ip_conntrack, 
regardless of the IP address of the interior interface.

Thanks for your help.

rtg

On Sunday 16 November 2003 02:52, Balazs Scheidler wrote:
>
> NAT mappings are established for NEW connections only, isn't it possible
> that your client sent an NTP request while your rule was not yet
> established? This means that it is entered the CONNTRACK table without the
> NAT manip and anything that comes later is not NATed as it is not a fresh
> CONNTRACK.
>
> Try filtering this traffic for 180secs and see it disappear from
> /proc/net/ip_conntrack, then remove the filtering and check whether it
> traversed the  nat/POSTROUTING chain.

-- 
Tim Gardner - timg@tpi.com 406-443-5357
TriplePoint, Inc. - http://www.tpi.com
PGP: http://www.tpi.com/PGP/Tim.txt

Re: 2.4 SNAT fails randomly

[Tim Gardner]

I've been thinking more about this scenario. When a standard distribution like 
SuSE-9.0 boots, the interfaces are  IFF_UP sometime before firewall rules are 
installed. This is particularly a problem in the case of a NAT router. Is 
there any way to flush /proc/net/ip_conntrack to make sure that you get rid 
of entries that were established between the time the interfaces were brought 
up and the time the rules were installed? Removing the ip_conntrack modules 
seems kind of brute force, and does not work on kernels where ip_conntrack is 
not a module.

rtg

On Sunday 16 November 2003 09:53, Tim Gardner wrote:
> Balazs,
>
> You are exactly correct. I swapped the interface creation statements with
> the iptables rules so that the rules were in place before any packets could
> be received from the interior net. In fact, I had to make sure that the
> interior net interface was down. Otherwise it would make an entry in
> ip_conntrack, regardless of the IP address of the interior interface.
>
> Thanks for your help.
>
> rtg
>
> On Sunday 16 November 2003 02:52, Balazs Scheidler wrote:
> > NAT mappings are established for NEW connections only, isn't it possible
> > that your client sent an NTP request while your rule was not yet
> > established? This means that it is entered the CONNTRACK table without
> > the NAT manip and anything that comes later is not NATed as it is not a
> > fresh CONNTRACK.
> >
> > Try filtering this traffic for 180secs and see it disappear from
> > /proc/net/ip_conntrack, then remove the filtering and check whether it
> > traversed the  nat/POSTROUTING chain.

-- 
Tim Gardner - timg@tpi.com
www.tpi.com 406-443-5357

Re: 2.4 SNAT fails randomly

[Balazs Scheidler]

On Tue, Nov 18, 2003 at 08:35:09AM -0700, Tim Gardner wrote:
> I've been thinking more about this scenario. When a standard distribution like 
> SuSE-9.0 boots, the interfaces are  IFF_UP sometime before firewall rules are 
> installed. This is particularly a problem in the case of a NAT router. Is 
> there any way to flush /proc/net/ip_conntrack to make sure that you get rid 
> of entries that were established between the time the interfaces were brought 
> up and the time the rules were installed? Removing the ip_conntrack modules 
> seems kind of brute force, and does not work on kernels where ip_conntrack is 
> not a module.

I think this scenario is inherently broken and the distribution should be
fixed. It is meant to be possible to load rulesets when the interfaces are
not yet configured.

Nevertheless the possibility to clear the conntrack table is desperately
needed as well as other conntrack manipulations. This area is getting less
neglected with the introduction of ctnetlink, albeit process is slow.

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1

Re: 2.4 SNAT fails randomly

[Eldad Zack]

On Tue, 18 Nov 2003, Tim Gardner wrote:

> I've been thinking more about this scenario. When a standard distribution like 
> SuSE-9.0 boots, the interfaces are  IFF_UP sometime before firewall rules are 
> installed. This is particularly a problem in the case of a NAT router. Is 
> there any way to flush /proc/net/ip_conntrack to make sure that you get rid 
> of entries that were established between the time the interfaces were brought 
> up and the time the rules were installed? Removing the ip_conntrack modules 
> seems kind of brute force, and does not work on kernels where ip_conntrack is 
> not a module.

There is a quick and dirty way to do it - forging a RST packet to the 
tuple you want to flush. I've used hping to forge such packets and 
iptstate to look at the conntrack table.

There really should be another way, though.



Eldad