UNTRACKED packets are identified as INVALID

UNTRACKED packets are identified as INVALID

[Sharon Tal]

Hi there,
I have 2 web servers behind an iptables based load-balancer, and I'm trying
to setup a graphics web server on the load-balancer, so that if a simple
static file is requested it will be able to respond instead of forwarding
the request to the web servers.
 
I've been trying to do that by matching packets at the raw table, setting
them to be UNTRACKED and leaving them on the LB.
The problem is that all UNTRACKED packets are identified as INVALID as soon
as they get to the mangle chain and dropped.
 
Here's my configuration (NS1, NS2 - the web servers, LB - the load balancer,
INTFACE/EXTFACE - internal/external interface)
<<--------------------------------------------------------------------------
---------------->>
echo 1 > /proc/sys/net/ipv4/ip_forward
 
$IPTABLES -F INPUT
$IPTABLES -F FORWARD
$IPTABLES -F OUTPUT
$IPTABLES -t raw  -F
$IPTABLES -t mangle -F
$IPTABLES -t nat -F
$IPTABLES -Z

# Drop invalid packets
$IPTABLES -t mangle -A PREROUTING -m state --state INVALID -j DROP
$IPTABLES -t mangle -A PREROUTING -p tcp ! --syn -m state --state NEW -j
DROP
 
# Prevent Spoofing
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -s 192.168.0.0/24 -p ! ICMP -j
DROP
 
# Prevent Smurf Attacks
$IPTABLES -t mangle -A PREROUTING -p icmp -i $EXTFACE -d 192.168.0.255 -j
DROP
 
# Prevent Syn-Flood, Ping-O-Death & Fast Port Scanning
$IPTABLES -t mangle -A PREROUTING -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
$IPTABLES -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
$IPTABLES -t mangle -A PREROUTING -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP

# Limit number of new connections/second from a single IP to a single port
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m state
--state NEW -m recent --name HTTP --set
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m state
--state NEW -m recent --name HTTP --update --seconds 2 --hitcount 100 -j
DROP

# Forward traffic from internal servers to the internet
$IPTABLES -t nat -A POSTROUTING -o $EXTFACE -j SNAT --to-source
192.168.100.153

# JPG requests should be answered by LB
$IPTABLES -t raw -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m string
--algo bm --string "GET /" -m string --algo bm --string ".jpg HTTP/1." -j
NOTRACK
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m state
--state UNTRACKED -j MARK --set-mark 0x11

# HTTP (25:75 Marking 1X)
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m mark
--mark 0x0 -m statistic --mode nth --every 4 -j MARK --set-mark 0x12
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m mark
--mark 0x0 -j MARK --set-mark 0x13
$IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -j RETURN
$IPTABLES -t nat -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m mark --mark
0x12 -j DNAT --to-destination $NS1IP:80
$IPTABLES -t nat -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m mark --mark
0x13 -j DNAT --to-destination $NS2IP:80
$IPTABLES -t nat -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m mark --mark
0x0 -j DNAT --to-destination $NS2IP:80
$IPTABLES -t nat -A PREROUTING -i $EXTFACE -p tcp --dport 80 -j RETURN
<<--------------------------------------------------------------------------
---------------->>
 
Any ideas on how to make this happen or what am I doing wrong?
 
Thanks!
 
Cheers!

Re: UNTRACKED packets are identified as INVALID

[Jan Engelhardt]

On Wednesday 2008-03-26 14:29, Sharon Tal wrote:
>
> Hi there,
> I have 2 web servers behind an iptables based load-balancer, and I'm trying
> to setup a graphics web server on the load-balancer, so that if a simple
> static file is requested it will be able to respond instead of forwarding
> the request to the web servers.
>
> I've been trying to do that by matching packets at the raw table, setting
> them to be UNTRACKED and leaving them on the LB.
> The problem is that all UNTRACKED packets are identified as INVALID as soon
> as they get to the mangle chain and dropped.

I would have a hard time reproducing it --

# iptables -t raw -A PREROUTING -s 134.76.13.21 -j NOTRACK
# iptables -t raw -A PREROUTING -s 134.76.13.21 -j LOGMARK
# iptables -t mangle -A PREROUTING -s 134.76.13.21 -j LOGMARK

# ping 134.76.13.21 -c1
PING 134.76.13.21 (134.76.13.21) 56(84) bytes of data.
[352431.272089] nfmark=0x0 secmark=0x0 classify=0x0 ct=UNTRACKED ctmark=NULL
ctstate=UNTRACKED ctstatus=NONE
[352431.272249] nfmark=0x0 secmark=0x0 classify=0x0 ct=UNTRACKED ctmark=NULL
ctstate=UNTRACKED ctstatus=NONE

# iptables -t mangle -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 145 packets, 115K bytes)
  pkts bytes target     prot opt in     out     source               destination

     1    84 LOGMARK    all  --  *      *       134.76.13.21         0.0.0.0/0
         LOGMARK level 4 prefix ""
     0     0            all  --  *      *       134.76.13.21         0.0.0.0/0
         ctstate INVALID

Now I am not sure what system you use, but this is my output on
kernel 2.6.23.

Re: UNTRACKED packets are identified as INVALID

[Pascal Hambourg]

Hello,

Sharon Tal a écrit :
> I have 2 web servers behind an iptables based load-balancer, and I'm trying
> to setup a graphics web server on the load-balancer, so that if a simple
> static file is requested it will be able to respond instead of forwarding
> the request to the web servers.
>  
> I've been trying to do that by matching packets at the raw table, setting
> them to be UNTRACKED and leaving them on the LB.
> The problem is that all UNTRACKED packets are identified as INVALID as soon
> as they get to the mangle chain and dropped.
> [...]
> # JPG requests should be answered by LB
> $IPTABLES -t raw -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m string
> --algo bm --string "GET /" -m string --algo bm --string ".jpg HTTP/1." -j
> NOTRACK
> $IPTABLES -t mangle -A PREROUTING -i $EXTFACE -p tcp --dport 80 -m state
> --state UNTRACKED -j MARK --set-mark 0x11

IMHO, the packets caught by the NOTRACK target are not the same as the 
packets in the INVALID state. The important thing is that only the 
individual packet containing the string will have the UNTRACKED state, 
not the whole connection. Allow me to guess what happens.

A SYN packet starting a new connection is received by the load-balancer. 
It contains no data, so it cannot match the NOTRACK rule. The conntrack 
creates a new entry for that connection. Then the packet - and the whole 
connection - is DNATed to one web server, and the conntrack entry is 
updated accordingly. This means that all subsequent incoming packets 
identified by the conntrack as belonging to that connection will be 
DNATed to the same web server.

The 3-way TCP handshake (SYN/ACK from the server, ACK from the client) 
completes. Then comes the packet carrying the HTTP request containing 
the string. This packet is UNTRACKED, so it is not DNATed and reaches 
the TCP layer of the load balancer. But it is not a SYN packet, and it 
does not match any local socket because the local TCP layer did not see 
the 3-way handshake. So it is dropped, maybe triggering a TCP RST packet 
back to the client (not sure about that).

I guess that the segment being UNTRACKED may disturb the TCP sequence 
number tracking, so at some point subsequent segments may be considered 
INVALID.

Anyway, NOTRACK won't do what you want. You can't change the destination 
of a whole TCP connection after it has started. I'm afraid that the only 
way to achieve what you want is to set up load balancing using a reverse 
proxy instead of NAT.

RE: UNTRACKED packets are identified as INVALID

[Sharon Tal]

 
Hi,

> Allow me to guess what happens.

> A SYN packet starting a new connection is received by the
load-balancer. 
> It contains no data, so it cannot match the NOTRACK rule. The
conntrack
> creates a new entry for that connection. Then the packet - and the
whole
> connection - is DNATed to one web server, and the conntrack entry is
updated
> accordingly. This means that all subsequent incoming packets
identified by the
> conntrack as belonging to that connection will be DNATed to the same
web server.

> The 3-way TCP handshake (SYN/ACK from the server, ACK from the client)
completes.
> Then comes the packet carrying the HTTP request containing the string.
This packet is
> UNTRACKED, so it is not DNATed and reaches the TCP layer of the load
balancer.
> But it is not a SYN packet, and it does not match any local socket
because the local
> TCP layer did not see the 3-way handshake. So it is dropped, maybe
triggering a TCP
> RST packet back to the client (not sure about that).

You are absolutely right (including the RST part).

Thanks.