Some weird issue with return traffic with redirect rule

Some weird issue with return traffic with redirect rule

[Pranav Desai]

Hello All,

We have a http proxy server running in transparent mode, with the
following rule to redirect port 80 traffic to the proxy port 8001.

1348K   81M REDIRECT   tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpts:80:82 redir ports 8001

We are running with iptables 1.3.8 and kernel 2.6.20.15

We are seeing cases where the return traffic is going out using the
proxyIP and port 8001, instead of using the origin servers IP and port
80. I have added the traces for both cases below. Most of the traffic
goes out correctly using the origin server IP and port, but the
traffic using port 8001 is not insignificant, hence we are a bit
concerned about it.

There are no connections coming in to port 8001.

I am not sure what could be causing this behavior or how I should go
about debugging this. Could the conntrack table or its usage be
screwed up?

I would appreciate any help I can get.

Thanks
-- Pranav


trace with traffic showing expected srcIP:srcPORT of the origin server
(80.93.57.77:80)
the 10.1.20.130 is the client.
======================================================
3:47:32.681258 IP 80.93.57.77.80 > 10.1.20.130.46236: P 1:511(510) ack
409 win 1716 <nop,nop,timestamp 21142119 3794704082>
13:47:32.681268 IP 80.93.57.77.80 > 10.1.20.130.46236: P 1:511(510)
ack 409 win 1716 <nop,nop,timestamp 21142119 3794704082>
13:47:32.766609 IP 80.93.57.77.80 > 10.1.20.130.2812: . ack 497 win
6432 <nop,nop,timestamp 2878779340 21142116>
13:47:32.787985 IP 81.176.228.45.80 > 10.1.20.130.13698: P 1:234(233)
ack 410 win 33304 <nop,nop,timestamp 3734311841 21142115>
13:47:32.788002 IP 10.1.20.130.13698 > 81.176.228.45.80: . ack 234 win
1716 <nop,nop,timestamp 21142145 3734311841>
13:47:32.788003 IP 10.1.20.130.13698 > 81.176.228.45.80: . ack 234 win
1716 <nop,nop,timestamp 21142145 3734311841>
13:47:32.797206 IP 10.1.20.130.13698 > 81.176.228.45.80: F 410:410(0)
ack 234 win 1716 <nop,nop,timestamp 21142148 3734311841>
13:47:32.797211 IP 10.1.20.130.13698 > 81.176.228.45.80: F 410:410(0)
ack 234 win 1716 <nop,nop,timestamp 21142148 3734311841>
13:47:32.797273 IP 81.176.228.45.80 > 10.1.20.130.50121: P 1:234(233)
ack 426 win 1716 <nop,nop,timestamp 21142148 3791907207>


Part of the trace which show traffic going out incorrectly using
proxyIP 10.10.224.5:8001
===========================================================
13:44:35.129072 IP 10.10.224.5.8001 > 10.1.20.130.36356: P
3807971279:3807971789(510) ack 897430549 win 1716 <nop,nop,timestamp
21097733 3542594707>
13:44:35.129079 IP 10.10.224.5.8001 > 10.1.20.130.36356: P 0:510(510)
ack 1 win 1716 <nop,nop,timestamp 21097733 3542594707>
13:44:35.158793 IP 10.10.224.5.8001 > 10.1.20.130.37781: P
3805867807:3805868040(233) ack 897312088 win 1716 <nop,nop,timestamp
21097741 3542594707>
13:44:35.158801 IP 10.10.224.5.8001 > 10.1.20.130.37781: P 0:233(233)
ack 1 win 1716 <nop,nop,timestamp 21097741 3542594707>
13:44:45.690977 IP 10.10.224.5.8001 > 10.1.20.130.49664: P
3804891276:3804891509(233) ack 898418728 win 1716 <nop,nop,timestamp
21100373 3542594707>

Re: Some weird issue with return traffic with redirect rule

[Grant Taylor]

On 10/17/08 01:30, Pranav Desai wrote:
> We are seeing cases where the return traffic is going out using the 
> proxyIP and port 8001, instead of using the origin servers IP and 
> port 80. I have added the traces for both cases below. Most of the 
> traffic goes out correctly using the origin server IP and port, but 
> the traffic using port 8001 is not insignificant, hence we are a bit 
> concerned about it.

*nod*

> There are no connections coming in to port 8001.

Ok...

I have to ask, is there a reason you are not configuring clients to talk 
directly to the proxy?  In my experience this works a lot better than 
transparent proxying.  I tend to use direct proxying as the primary 
method and then transparent proxying as a backup and to catch devices 
that don't know how to talk to a proxy.

> I am not sure what could be causing this behavior or how I should go 
> about debugging this. Could the conntrack table or its usage be 
> screwed up?

Are there any DMESG or syslog entries about the connection table being full?



Grant. . . .

Re: Some weird issue with return traffic with redirect rule

[Pranav Desai]

On Fri, Oct 17, 2008 at 10:19 AM, Grant Taylor
<gtaylor@riverviewtech.net> wrote:
> On 10/17/08 01:30, Pranav Desai wrote:
>>
>> We are seeing cases where the return traffic is going out using the
>> proxyIP and port 8001, instead of using the origin servers IP and port 80. I
>> have added the traces for both cases below. Most of the traffic goes out
>> correctly using the origin server IP and port, but the traffic using port
>> 8001 is not insignificant, hence we are a bit concerned about it.
>
> *nod*
>
>> There are no connections coming in to port 8001.
>
> Ok...
>
> I have to ask, is there a reason you are not configuring clients to talk
> directly to the proxy?  In my experience this works a lot better than
> transparent proxying.  I tend to use direct proxying as the primary method
> and then transparent proxying as a backup and to catch devices that don't
> know how to talk to a proxy.
>

Too many clients will have to change their settings. Not feasible in our case.

>> I am not sure what could be causing this behavior or how I should go about
>> debugging this. Could the conntrack table or its usage be screwed up?
>
> Are there any DMESG or syslog entries about the connection table being full?
>

There is no info there, and the tables are not getting full. Here are
the conntrack settings.

net.ipv4.ip_conntrack_max = 1048576
net.ipv4.netfilter.ip_conntrack_buckets = 1048576
net.ipv4.netfilter.ip_conntrack_count = 63908
net.ipv4.netfilter.ip_conntrack_max = 1048576

-- Pranav

>
>
> Grant. . . .
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Some weird issue with return traffic with redirect rule

[Grant Taylor]

On 10/17/08 14:46, Pranav Desai wrote:
> Too many clients will have to change their settings. Not feasible in 
> our case.

*nod*

This is where auto-configure scripts come in to play.

If you can't, you cant.  No point in ruffling any feathers over it.  If 
transparent proxying is working for you then go for it.

> There is no info there, and the tables are not getting full. Here are 
> the conntrack settings.
> 
> net.ipv4.ip_conntrack_max = 1048576
> net.ipv4.netfilter.ip_conntrack_buckets = 1048576
> net.ipv4.netfilter.ip_conntrack_count = 63908
> net.ipv4.netfilter.ip_conntrack_max = 1048576

If conntrack is not getting full I wonder if some packets are 
accidentally not being associated and thus not being handled correctly.

Dare I say it, you may be looking at setting up TCPDump (or the likes) 
to record all packets.  That way when you do have packets that did not 
get handled correctly you can go back and look at the rest of the 
packets that should have been associated but were not.



Grant. . . .

Re: Some weird issue with return traffic with redirect rule

[Pranav Desai]

On Mon, Oct 20, 2008 at 2:24 PM, Grant Taylor <gtaylor@riverviewtech.net> wrote:
> On 10/17/08 14:46, Pranav Desai wrote:
>>
>> Too many clients will have to change their settings. Not feasible in our
>> case.
>
> *nod*
>
> This is where auto-configure scripts come in to play.
>
> If you can't, you cant.  No point in ruffling any feathers over it.  If
> transparent proxying is working for you then go for it.
>
>> There is no info there, and the tables are not getting full. Here are the
>> conntrack settings.
>>
>> net.ipv4.ip_conntrack_max = 1048576
>> net.ipv4.netfilter.ip_conntrack_buckets = 1048576
>> net.ipv4.netfilter.ip_conntrack_count = 63908
>> net.ipv4.netfilter.ip_conntrack_max = 1048576
>
> If conntrack is not getting full I wonder if some packets are accidentally
> not being associated and thus not being handled correctly.
>
> Dare I say it, you may be looking at setting up TCPDump (or the likes) to
> record all packets.  That way when you do have packets that did not get
> handled correctly you can go back and look at the rest of the packets that
> should have been associated but were not.
>

I did. I can attach the raw trace as well, but not sure if the mailer
accepts it. So describing it here.

Just to reiterate the setup.
Its a transparent proxy with tproxy. The redirect rule on the
client-side redirects all port 80 traffic to port 8001 which is my
proxy port.
Proxy IP: 10.10.224.6
Origin Server: 81.176.228.109
Client IP: 10.1.21.202

As you can see the client makes the connection, sends the GET request
(pkt 3). The Proxy ACKs with the origin server IP.
Sends the HTTP response (pkt 5). Retransmits it (pkt 6 and 7) not sure
why the time period between the retransmits is that large. I was
expecting a to be much smaller. Anyway, maybe someone here has an idea
why its that way.

Then the client sends a FIN and the proxy ACKs it, and then the last
pkt is the problem pkt where the proxy sends it out using it own IP
and port.

So I am guessing that the conntrack expires by then and the pkt goes
out without any change ? Any thoughts ?

There are other cases where the FIN,ACK is retransmitted multiple time
and the last one in them is with src port 8001.

The client-proxy connection is a radio link, hence the crappy performance.

-- Pranav

=====> trace <====
13:00:40.468612 IP 10.1.21.202.42472 > 81.176.228.109.80: S
260637943:260637943(0) win 48960 <mss 1360,nop,nop,timestamp
3503521357 0,nop,wscale 0,nop,nop,sackOK>
13:00:40.468631 IP 81.176.228.109.80 > 10.1.21.202.42472: S
1982162570:1982162570(0) ack 260637944 win 5792 <mss
1460,sackOK,timestamp 128429945 3503521357,nop,wscale 2>
13:00:46.809934 IP 10.1.21.202.42472 > 81.176.228.109.80: P
260637944:260639005(1061) ack 1982162571 win 48960 <nop,nop,timestamp
3509490107 128429945>
13:00:46.809955 IP 81.176.228.109.80 > 10.1.21.202.42472: . ack
260639005 win 1979 <nop,nop,timestamp 128431530 3509490107>
13:00:50.202386 IP 81.176.228.109.80 > 10.1.21.202.42472: P
1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
128432378 3509490107>
13:01:04.897471 IP 81.176.228.109.80 > 10.1.21.202.42472: P
1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
128436052 3509490107>
13:01:34.292153 IP 81.176.228.109.80 > 10.1.21.202.42472: P
1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
128443400 3509490107>
13:01:59.589460 IP 10.1.21.202.42472 > 81.176.228.109.80: F
260639005:260639005(0) ack 1982162571 win 48960 <nop,nop,timestamp
3582324982 128431530>
13:01:59.589493 IP 81.176.228.109.80 > 10.1.21.202.42472: F
1982163119:1982163119(0) ack 260639006 win 1979 <nop,nop,timestamp
128449724 3582324982>
13:02:33.077175 IP 10.10.224.6.8001 > 10.1.21.202.42472: P
1982162571:1982163119(548) ack 260639006 win 1979 <nop,nop,timestamp
128458096 3582324982>
=====> trace <====

>
>
> Grant. . . .
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Some weird issue with return traffic with redirect rule

[Pranav Desai]

The issue is related to the conntrack timeouts and tcp retries.
Thought of updating the thread, in case anyone else is looking.

Here is how you can reproduce it ...

* Set the conntrack last ack timeout to something low and the retries1
to high, so force this behavior

sysctl -w net.ipv4.tcp_retries1=10
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack=15

* Add a redirect rule to the server
 iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8001
* make a connection to a server on port 80 which has a redirect rule.
* send some data to make sure everything is ok.
* Add iptables drop rules on the client to block anything to and from
the server.

iptables -A OUTPUT -p tcp -d <serverIP> -o eth1 -j DROP
iptables -A INPUT -p tcp -s <serverIP> -i eth1 -j DROP

* close the server.
* You should see a bunch of FINs going out
* Also check the conntrack entry for the connection, it will be in
LAST_ACK state.
* Any remaining FINs (out of 10) that goes out after the entry has
been removed, will go out with the original port and not the redirect
port.

So, as a solution you have to make sure that the conntrack entry stays
as long as TCP is going send data.
In my case, the one originally mentioned in the thread, the network is
quite bad, so the retransmit interval is quite high, as a result of
which the conntrack entry expires and the pkts are sent out with wrong
src port.

-- Pranav

On Mon, Oct 20, 2008 at 7:02 PM, Pranav Desai <pranavadesai@gmail.com> wrote:
> On Mon, Oct 20, 2008 at 2:24 PM, Grant Taylor <gtaylor@riverviewtech.net> wrote:
>> On 10/17/08 14:46, Pranav Desai wrote:
>>>
>>> Too many clients will have to change their settings. Not feasible in our
>>> case.
>>
>> *nod*
>>
>> This is where auto-configure scripts come in to play.
>>
>> If you can't, you cant.  No point in ruffling any feathers over it.  If
>> transparent proxying is working for you then go for it.
>>
>>> There is no info there, and the tables are not getting full. Here are the
>>> conntrack settings.
>>>
>>> net.ipv4.ip_conntrack_max = 1048576
>>> net.ipv4.netfilter.ip_conntrack_buckets = 1048576
>>> net.ipv4.netfilter.ip_conntrack_count = 63908
>>> net.ipv4.netfilter.ip_conntrack_max = 1048576
>>
>> If conntrack is not getting full I wonder if some packets are accidentally
>> not being associated and thus not being handled correctly.
>>
>> Dare I say it, you may be looking at setting up TCPDump (or the likes) to
>> record all packets.  That way when you do have packets that did not get
>> handled correctly you can go back and look at the rest of the packets that
>> should have been associated but were not.
>>
>
> I did. I can attach the raw trace as well, but not sure if the mailer
> accepts it. So describing it here.
>
> Just to reiterate the setup.
> Its a transparent proxy with tproxy. The redirect rule on the
> client-side redirects all port 80 traffic to port 8001 which is my
> proxy port.
> Proxy IP: 10.10.224.6
> Origin Server: 81.176.228.109
> Client IP: 10.1.21.202
>
> As you can see the client makes the connection, sends the GET request
> (pkt 3). The Proxy ACKs with the origin server IP.
> Sends the HTTP response (pkt 5). Retransmits it (pkt 6 and 7) not sure
> why the time period between the retransmits is that large. I was
> expecting a to be much smaller. Anyway, maybe someone here has an idea
> why its that way.
>
> Then the client sends a FIN and the proxy ACKs it, and then the last
> pkt is the problem pkt where the proxy sends it out using it own IP
> and port.
>
> So I am guessing that the conntrack expires by then and the pkt goes
> out without any change ? Any thoughts ?
>
> There are other cases where the FIN,ACK is retransmitted multiple time
> and the last one in them is with src port 8001.
>
> The client-proxy connection is a radio link, hence the crappy performance.
>
> -- Pranav
>
> =====> trace <====
> 13:00:40.468612 IP 10.1.21.202.42472 > 81.176.228.109.80: S
> 260637943:260637943(0) win 48960 <mss 1360,nop,nop,timestamp
> 3503521357 0,nop,wscale 0,nop,nop,sackOK>
> 13:00:40.468631 IP 81.176.228.109.80 > 10.1.21.202.42472: S
> 1982162570:1982162570(0) ack 260637944 win 5792 <mss
> 1460,sackOK,timestamp 128429945 3503521357,nop,wscale 2>
> 13:00:46.809934 IP 10.1.21.202.42472 > 81.176.228.109.80: P
> 260637944:260639005(1061) ack 1982162571 win 48960 <nop,nop,timestamp
> 3509490107 128429945>
> 13:00:46.809955 IP 81.176.228.109.80 > 10.1.21.202.42472: . ack
> 260639005 win 1979 <nop,nop,timestamp 128431530 3509490107>
> 13:00:50.202386 IP 81.176.228.109.80 > 10.1.21.202.42472: P
> 1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
> 128432378 3509490107>
> 13:01:04.897471 IP 81.176.228.109.80 > 10.1.21.202.42472: P
> 1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
> 128436052 3509490107>
> 13:01:34.292153 IP 81.176.228.109.80 > 10.1.21.202.42472: P
> 1982162571:1982163119(548) ack 260639005 win 1979 <nop,nop,timestamp
> 128443400 3509490107>
> 13:01:59.589460 IP 10.1.21.202.42472 > 81.176.228.109.80: F
> 260639005:260639005(0) ack 1982162571 win 48960 <nop,nop,timestamp
> 3582324982 128431530>
> 13:01:59.589493 IP 81.176.228.109.80 > 10.1.21.202.42472: F
> 1982163119:1982163119(0) ack 260639006 win 1979 <nop,nop,timestamp
> 128449724 3582324982>
> 13:02:33.077175 IP 10.10.224.6.8001 > 10.1.21.202.42472: P
> 1982162571:1982163119(548) ack 260639006 win 1979 <nop,nop,timestamp
> 128458096 3582324982>
> =====> trace <====
>
>>
>>
>> Grant. . . .
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>