netfilter resets TCP conversation that was DNATed from the local machine to another

netfilter resets TCP conversation that was DNATed from the local machine to another

[Michael]

Dear netfilter gods,

I have a configuration, so:

/------------\ .0.2     .{0,1}.1 /----------\ 1.2.3.4  (          )
| Web server |-----+-------------| firewall |---------(  Internet  )
\------------/     |        eth0 |  Squid   | eth1     (          )
                    |             \----------/
/---------\ .1.2   |
| browser |--------/
\---------/

- The 192.168.{0,1}. subnets run on the same wire.
- Port 80 on the public i/f is DNATed to the internal Web server.

The firewall is running Squid to proxy for 192.168.1. clients, and it 
works fine *except* when the target server resolves to a public IP on 
eth1.  When that happens, I see the client-to-Squid communication go OK, 
then Squid send a SYN (from .0.1) to .0.2:80, .0.2 sends a SYN ACK,... 
but then netfilter spontaneously issues a RST to .0.2:80 from another 
port (i.e., not the one that Squid was using)!  I have no 
reject-with-tcp-reset lines in my tables.

What up?


Squid really doesn't belong on a firewall, but I'm curious to resolve 
this mystery first.

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Arnt Karlsen]

On Fri, 27 Jun 2003 20:50:58 -0700, 
Michael <freeware@adsl-209-204-165-151.sonic.net> wrote in message 
<3EFD10A2.6020807@adsl-209-204-165-151.sonic.net>:

> Dear netfilter gods,
> 
> I have a configuration, so:
> 
> /------------\ .0.2     .{0,1}.1 /----------\ 1.2.3.4  (          )
> | Web server |-----+-------------| firewall |---------(  Internet  )
> \------------/     |        eth0 |  Squid   | eth1     (          )
>                     |             \----------/
> /---------\ .1.2   |
> | browser |--------/
> \---------/
> 
> - The 192.168.{0,1}. subnets run on the same wire.

..with what subnet?  /23 oughtta work.

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Ramin Dousti]

On Fri, Jun 27, 2003 at 08:50:58PM -0700, Michael wrote:

> Dear netfilter gods,
> 
> I have a configuration, so:
> 
> /------------\ .0.2     .{0,1}.1 /----------\ 1.2.3.4  (          )
> | Web server |-----+-------------| firewall |---------(  Internet  )
> \------------/     |        eth0 |  Squid   | eth1     (          )
>                    |             \----------/
> /---------\ .1.2   |
> | browser |--------/
> \---------/
> 
> - The 192.168.{0,1}. subnets run on the same wire.
> - Port 80 on the public i/f is DNATed to the internal Web server.
> 
> The firewall is running Squid to proxy for 192.168.1. clients, and it 
> works fine *except* when the target server resolves to a public IP on 
> eth1.  When that happens, I see the client-to-Squid communication go OK, 
> then Squid send a SYN (from .0.1) to .0.2:80, .0.2 sends a SYN ACK,... 
> but then netfilter spontaneously issues a RST to .0.2:80 from another 
> port (i.e., not the one that Squid was using)!

No idea why this RST is being sent (might have to do with your rule set or
more possibly the internals of squid) but the fact that you say the RST
sending port is not the same as the initiating SYN port should not break
anything. Can you confirm this?

Ramin

> I have no reject-with-tcp-reset lines in my tables.
> 
> What up?
> 
> 
> Squid really doesn't belong on a firewall, but I'm curious to resolve 
> this mystery first.
> 

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

What would you like me to confirm? That it's broken? It is broken. That 
the RST sending port is not the same as the initiating SYN port? It is 
not; it's low, just above 1024, so I assumed it to be 
netfilter-generated, whereas the Squid port was in the 30000 range. That 
my rule set isn't sending it? "I have no reject-with-tcp-reset lines in 
my tables." Don't know what else you could mean.

Ramin Dousti wrote:

>>I have a configuration, so:
>>
>>/------------\ .0.2     .{0,1}.1 /----------\ 1.2.3.4  (          )
>>| Web server |-----+-------------| firewall |---------(  Internet  )
>>\------------/     |        eth0 |  Squid   | eth1     (          )
>>                   |             \----------/
>>/---------\ .1.2   |
>>| browser |--------/
>>\---------/
>>
>>- The 192.168.{0,1}. subnets run on the same wire.
>>- Port 80 on the public i/f is DNATed to the internal Web server.
>>
>>The firewall is running Squid to proxy for 192.168.1. clients, and it 
>>works fine *except* when the target server resolves to a public IP on 
>>eth1.  When that happens, I see the client-to-Squid communication go OK, 
>>then Squid send a SYN (from .0.1) to .0.2:80, .0.2 sends a SYN ACK,... 
>>but then netfilter spontaneously issues a RST to .0.2:80 from another 
>>port (i.e., not the one that Squid was using)!
>>    
>>
>
>No idea why this RST is being sent (might have to do with your rule set or
>more possibly the internals of squid) but the fact that you say the RST
>sending port is not the same as the initiating SYN port should not break
>anything. Can you confirm this?
>
>Ramin
>
>  
>
>>I have no reject-with-tcp-reset lines in my tables.
>>

[-- Attachment #2: Type: text/html, Size: 2059 bytes --]

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Alistair Tonner]

On June 29, 2003 10:22 pm, Michael wrote:
> What would you like me to confirm? That it's broken? It is broken. That
> the RST sending port is not the same as the initiating SYN port? It is
> not; it's low, just above 1024, so I assumed it to be
> netfilter-generated, whereas the Squid port was in the 30000 range. That
> my rule set isn't sending it? "I have no reject-with-tcp-reset lines in
> my tables." Don't know what else you could mean.
>
> Ramin Dousti wrote:
> >>I have a configuration, so:
> >>
> >>/------------\ .0.2     .{0,1}.1 /----------\ 1.2.3.4  (          )
> >>
> >>| Web server |-----+-------------| firewall |---------(  Internet  )
> >>
> >>\------------/     |        eth0 |  Squid   | eth1     (          )
> >>
> >>                   |             \----------/
> >>
> >>/---------\ .1.2   |
> >>
> >>| browser |--------/
> >>
> >>\---------/
> >>
> >>- The 192.168.{0,1}. subnets run on the same wire.
> >>- Port 80 on the public i/f is DNATed to the internal Web server.
> >>
> >>The firewall is running Squid to proxy for 192.168.1. clients, and it
> >>works fine *except* when the target server resolves to a public IP on
> >>eth1.  When that happens, I see the client-to-Squid communication go OK,
> >>then Squid send a SYN (from .0.1) to .0.2:80, .0.2 sends a SYN ACK,...
> >>but then netfilter spontaneously issues a RST to .0.2:80 from another
> >>port (i.e., not the one that Squid was using)!
> >
> >No idea why this RST is being sent (might have to do with your rule set or
> >more possibly the internals of squid) but the fact that you say the RST
> >sending port is not the same as the initiating SYN port should not break
> >anything. Can you confirm this?
> >
> >Ramin
> >
> >>I have no reject-with-tcp-reset lines in my tables.


	Ummm:
	     I wonder.
	     Does the squid to webserver connection ever go through *any* 
firewalling?
	Does the webserver to squid response get handled by any firewalling?

	I *suspect* that squid may be avoiding the firewall, sending connection 
	direct to webserver, but reply from webserver is hitting the firewall, and
	since it isn't thus in any conntrack, getting b0rked.  Mind you... 
	I'd need to see the ruleset first.  Question comes as ... where does 
	*squid* resolve names and what routes will it use? 

	To answer this at all I'd need to see the full ruleset.

	Then again ... I might be way off on this, but I've seen something remotely 	
	similar in practice where the hosts file on the firewall server mislead 
squid.

-- 

	Alistair Tonner
	nerdnet.ca
	Senior Systems Analyst - RSS
	
     Any sufficiently advanced technology will have the appearance of magic.
	Lets get magical!

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Ramin Dousti]

On Sun, Jun 29, 2003 at 07:22:13PM -0700, Michael wrote:

> What would you like me to confirm? That it's broken? It is broken. That 
> the RST sending port is not the same as the initiating SYN port? It is 
> not; it's low, just above 1024, so I assumed it to be 
> netfilter-generated, whereas the Squid port was in the 30000 range. That 
> my rule set isn't sending it? "I have no reject-with-tcp-reset lines in 
> my tables." Don't know what else you could mean.

What I mean is this:

squid:		ip1:port1
webserver:	ip2:80

Then what you say is:

packet1:	ip1:port1 -> ip2:80    (SYN)
packet2:	ip2:80    -> ip1:port1 (SYN ACK)
packet3:	ip1:port2 -> ip2:80    (RST)

What I'm saying is that the third packet should not be able to tear down
the connection between ip1:port1 <-> ip2:80  just because the ports are
different. The TCP stack on ip2 should discard this RST packet...

Again, my question:
do you see any other packets between ip1:port1 <-> ip2:port2 after the RST
packet?

Ramin

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Ray Leach]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

This is an IE browser problem.
Normal browsers don't send RST after the connection has ended.

On Mon, 2003-06-30 at 16:29, Ramin Dousti wrote:
> On Sun, Jun 29, 2003 at 07:22:13PM -0700, Michael wrote:
> 
> > What would you like me to confirm? That it's broken? It is broken. That 
> > the RST sending port is not the same as the initiating SYN port? It is 
> > not; it's low, just above 1024, so I assumed it to be 
> > netfilter-generated, whereas the Squid port was in the 30000 range. That 
> > my rule set isn't sending it? "I have no reject-with-tcp-reset lines in 
> > my tables." Don't know what else you could mean.
> 
> What I mean is this:
> 
> squid:		ip1:port1
> webserver:	ip2:80
> 
> Then what you say is:
> 
> packet1:	ip1:port1 -> ip2:80    (SYN)
> packet2:	ip2:80    -> ip1:port1 (SYN ACK)
> packet3:	ip1:port2 -> ip2:80    (RST)
> 
> What I'm saying is that the third packet should not be able to tear down
> the connection between ip1:port1 <-> ip2:80  just because the ports are
> different. The TCP stack on ip2 should discard this RST packet...
> 
> Again, my question:
> do you see any other packets between ip1:port1 <-> ip2:port2 after the RST
> packet?
> 
> Ramin
> 
> 
> 
-- 
--
Raymond Leach <raymondl@knowledgefactory.co.za>
Network Support Specialist
http://www.knowledgefactory.co.za
"lynx -source http://www.rchq.co.za/raymondl.asc | gpg --import"
Key fingerprint = 7209 A695 9EE0 E971 A9AD  00EE 8757 EE47 F06F FB28
--

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Ramin Dousti]

On Mon, Jun 30, 2003 at 04:44:00PM +0200, Ray Leach wrote:

> This is an IE browser problem.
> Normal browsers don't send RST after the connection has ended.

He hasn't said anything about a browser running on his squid machine (yet).
And besides I don't think you have IE for linux, do you?
My whole point is the RST from a different port is annoying/weird but
harmless to that particular connection.

My feeling is that he's providing just part of the facts...

Ramin

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Michael]

Ramin Dousti wrote:

>What I mean is this:
>
>squid:		ip1:port1
>webserver:	ip2:80
>
>Then what you say is:
>
>packet1:	ip1:port1 -> ip2:80    (SYN)
>packet2:	ip2:80    -> ip1:port1 (SYN ACK)
>packet3:	ip1:port2 -> ip2:80    (RST)
>
>What I'm saying is that the third packet should not be able to tear down
>the connection between ip1:port1 <-> ip2:80  just because the ports are
>different. The TCP stack on ip2 should discard this RST packet...
>
>Again, my question:
>do you see any other packets between ip1:port1 <-> ip2:port2 after the RST
>packet?
>
I see. Here's the configuration, again, for reference:

/------------\ .0.{8,9}     .{0,1}.1 /----------\ 1.2.3.{4,5,6}  (          )
| Web server |---------+-------------| firewall |---------------(  Internet  )
\------------/         |        eth0 |  Squid   | eth1           (          )
                       |             \----------/
/---------\ .1.2       |
| browser |------------/
\---------/

I lied earlier for simplicity. I have multiple addresses on eth1 and on 
the Web server.

I stop and start Squid. I access my domain, xxx.org, from my browser. On 
eth0, I see

10:52:42.684703 192.168.1.2.4358 > 192.168.1.1.squid: S 3347635646:3347635646(0) win 16060 <mss 1460,sackOK,timestamp 54041369 0,nop,wscale 0> (DF)
10:52:42.685601 192.168.1.1.squid > 192.168.1.2.4358: S 3951034191:3951034191(0) ack 3347635647 win 5792 <mss 1460,sackOK,timestamp 77755654 54041369,nop,wscale 0> (DF)
10:52:42.685952 192.168.1.2.4358 > 192.168.1.1.squid: . ack 1 win 16060 <nop,nop,timestamp 54041369 77755654> (DF)
10:52:42.686482 192.168.1.2.4358 > 192.168.1.1.squid: P 1:432(431) ack 1 win 16060 <nop,nop,timestamp 54041369 77755654> (DF)
10:52:42.686801 192.168.1.1.squid > 192.168.1.2.4358: . ack 432 win 6432 <nop,nop,timestamp 77755655 54041369> (DF)

That's my browser querying Squid. On eth1, I see

10:52:42.690711 1.2.3.4.32804 > myisp.domain: 2+ A? xxx.org. [|domain] (DF)
10:52:42.691228 1.2.3.4.32804 > myisp.domain: 3+ A? xxx.org. [|domain] (DF)
10:52:42.710447 myisp.domain > 1.2.3.4.32804: 2* 1/2/2 xxx.org. A 1.2.3.5 (119) (DF)
10:52:42.715838 myisp.domain > 1.2.3.4.32804: 3* 1/2/2 xxx.org. A 1.2.3.5 (119) (DF)

That's Squid looking up my domain. (Why twice? I don't know.) The OUTPUT 
chain in the nat table is

Chain OUTPUT (policy ACCEPT)
 target prot opt in out source     destination         
 DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.5     multiport dports 80,443 to:192.168.0.8 
 DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.6     multiport dports 80,443 to:192.168.0.9 

Then on eth0, I see

10:52:42.719385 192.168.0.1.36065 > 192.168.0.8.www: S 3950921369:3950921369(0) win 32767 <mss 16396,sackOK,timestamp 77755658 0,nop,wscale 0> (DF)
10:52:42.719797 192.168.0.8.www > 192.168.0.1.36065: S 3348203817:3348203817(0) ack 3950921370 win 5792 <mss 1460,sackOK,timestamp 30595894 77755658,nop,wscale 0> (DF)
10:52:42.720206 192.168.0.1.1028 > 192.168.0.8.www: R 3950921370:3950921370(0) win 0 (DF)
10:52:45.716310 192.168.0.1.36065 > 192.168.0.8.www: S 3950921369:3950921369(0) win 32767 <mss 16396,sackOK,timestamp 77755958 0,nop,wscale 0> (DF)
10:52:45.716595 192.168.0.8.www > 192.168.0.1.36065: S 3348203817:3348203817(0) ack 3950921370 win 5792 <mss 1460,sackOK,timestamp 30596194 77755658,nop,wscale 0> (DF)
10:52:45.716974 192.168.0.1.1028 > 192.168.0.8.www: R 3950921370:3950921370(0) win 0 (DF)
10:52:46.910244 192.168.0.8.www > 192.168.0.1.36065: S 3348203817:3348203817(0) ack 3950921370 win 5792 <mss 1460,sackOK,timestamp 30596314 77755658,nop,wscale 0> (DF)
10:52:46.910653 192.168.0.1.1028 > 192.168.0.8.www: R 3950921370:3950921370(0) win 0 (DF)

and this pattern repeats, with Squid resending its SYN on port 36065, 
and the server asynchronously resending its SYN-ACK, and netfilter (or 
something) sending a RST for every SYN-ACK and thinking it's handled the 
incoming packet. So, no, the connection isn't effectively torn down.

Eventually Squid changes to port 36066, with the same effect, then sends 
its error page back to the browser. That's all the traffic that I see, 
excepting arp, igmp, my own ssh, and an external www request that 
happened during the long wait.

Alistair Tonner wrote:

>Does the squid to webserver connection ever go through *any* firewalling?
>
Yes. I see the packet count incremented (for the first packet) in the 
above OUTPUT chain to .0.8. I also see a line in /proc/net/ip_conntrack like

tcp 6 58 SYN_RECV src=1.2.3.5 dst=1.2.3.5 sport=36065 dport=80 src=192.168.0.8 dst=192.168.0.1 sport=80 dport=36065 use=1 

Curious that it's a SYN_RECV and not a SYN_SENT.

>Does the webserver to squid response get handled by any firewalling?
>
It seems to. I get packets counted in the appropriate ACCEPT rules.


Thanks for thinking about this.

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Alistair Tonner]

On June 30, 2003 04:07 pm, Michael wrote:

>
> That's Squid looking up my domain. (Why twice? I don't know.) The OUTPUT
> chain in the nat table is
>
> Chain OUTPUT (policy ACCEPT)
>  target prot opt in out source     destination
>  DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.5     multiport dports 80,443
> to:192.168.0.8 DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.6     multiport
> dports 80,443 to:192.168.0.9
>
>

	Please read this page
	
	http://www.netfilter.org/documentation/HOWTO//NAT-HOWTO-3.html

	Why are you DNATting in OUTPUT?
-- 

	Alistair Tonner
	nerdnet.ca
	Senior Systems Analyst - RSS
	
     Any sufficiently advanced technology will have the appearance of magic.
	Lets get magical!

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Ramin Dousti]

On Tue, Jul 01, 2003 at 02:00:00AM -0400, Alistair Tonner wrote:

> On June 30, 2003 04:07 pm, Michael wrote:
> 
> >
> > That's Squid looking up my domain. (Why twice? I don't know.) The OUTPUT
> > chain in the nat table is
> >
> > Chain OUTPUT (policy ACCEPT)
> >  target prot opt in out source     destination
> >  DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.5     multiport dports 80,443
> > to:192.168.0.8 DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.6     multiport
> > dports 80,443 to:192.168.0.9
> >
> >
> 
> 	Please read this page
> 	
> 	http://www.netfilter.org/documentation/HOWTO//NAT-HOWTO-3.html
> 
> 	Why are you DNATting in OUTPUT?

Good catch. I guess he's doing this because squid is bound to the public
IP's. However, this is not necessary and for older iptables versions there
was a problem with natting in the OUTPUT chain...

Ramin

> -- 
> 
> 	Alistair Tonner
> 	nerdnet.ca
> 	Senior Systems Analyst - RSS
> 	
>      Any sufficiently advanced technology will have the appearance of magic.
> 	Lets get magical!

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Michael]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

Alistair Tonner wrote:

>On June 30, 2003 04:07 pm, Michael wrote:
>  
>
>>The OUTPUT chain in the nat table is
>>
>>Chain OUTPUT (policy ACCEPT)
>> target prot opt in out source     destination
>> DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.5     multiport dports 80,443
>>to:192.168.0.8 DNAT   tcp  --  *  *   0.0.0.0/0  1.2.3.6     multiport
>>dports 80,443 to:192.168.0.9
>>    
>>
>	Please read this page
>	
>	http://www.netfilter.org/documentation/HOWTO//NAT-HOWTO-3.html
>
I've read it before. How is it relevant?

>	Why are you DNATting in OUTPUT?
>
Because the packets are originating on the firewall, from Squid. Squid 
thinks xxx.org is at 1.2.3.5, so DNAT is needed to change that to 
192.168.0.8.

I also have identical rules in the PREROUTING chain of the nat table, if 
you were wondering. Those rules work for requests from the Internet.


[-- Attachment #2: Type: text/html, Size: 1577 bytes --]

Re: netfilter resets TCP conversation that was DNATed from the local machine to another

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Alistair Tonner wrote:

>On July 1, 2003 12:24 pm, Michael wrote:
>  
>
>>Alistair Tonner wrote:
>>    
>>
>>>	Why are you DNATting in OUTPUT?
>>>      
>>>
>>Because the packets are originating on the firewall, from Squid. Squid
>>thinks xxx.org is at 1.2.3.5, so DNAT is needed to change that to
>>192.168.0.8.
>>
>>I also have identical rules in the PREROUTING chain of the nat table, if
>>you were wondering. Those rules work for requests from the Internet.
>>    
>>
>	Okay .. .after reading up some on 1.2.8 ... that *should* 
>	work.  BUT .. to me it seems evil ... I'm forever bound by 
>	the concept that mangling the destination (or source for that matter)
>	should be done before or after the basic network routing has been done.
>
Routing is done to locally-generated packets after OUTPUT. See 
http://www.netfilter.org/documentation/HOWTO//netfilter-hacking-HOWTO-3.html

>	it seems unessesary to me for this reason:
>
>	squid calls to the resolved ip (your outside ip) ...which should route
>	out of the firewall, and back to its outside IP .. then the PREROUTING 
>	rules should catch these packets and reroute them correctly,
>	then the RETURN path would make sense.  It seems to me that 
>	DNATTING directly in output to force the packets out to the 
>	inside IP shortcuts things, and then breaks the return path.
>
If I remove the nat table's OUTPUT chain rules, I get a "connection 
refused", because there is no Web server running on 1.2.3.5 port 80. 
Locally-generated packets do not go through PREROUTING. They don't go 
"out of the firewall" unless they get routed there, and if that happens, 
they aren't coming back unless another machine returns them.

I'll post to LARTC.


[-- Attachment #2: Type: text/html, Size: 2393 bytes --]