Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[James MacLean]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

Hi Folks,

Today we noticed that some traffic appears to be getting by netfilter 
unNATed. The computers use typical SNATing such as :

iptables -t nat -I POSTROUTING -j SNAT -s <private ip space> -o eth0 
--to <IP of eth0>

While we were dumping traffic on the public interface, we noticed 
private IPs showing up in our dumps.

We verified that there existed sessions between private IPs and public 
Internet sites where some of the private traffic was returning to the 
Internet site not NATed.

We started watching, and saw this occur on servers with 2.6.11 and 
2.6.10 kernels which are all we have to test with.

These kernels were compiled locally.

For example, tcpdumping on a public interface where no private IPs 
should live :

15:19:22.066190 IP 10.0.7.100.2640 > 142.176.33.171.http: R 
2946695981:2946695981(0) ack 4138631275 win 0
15:19:22.066516 IP 10.0.7.100.2641 > 142.176.33.170.http: R 
818314876:818314876(0) ack 4144245614 win 0
15:19:22.066684 IP 10.0.7.100.2638 > 142.176.33.170.http: R 
3381617258:3381617258(0) ack 4136231656 win 0
15:19:22.067353 IP 10.0.7.100.2639 > 142.176.33.170.http: R 
3920569986:3920569986(0) ack 4133397861 win 0
15:19:22.068045 IP 10.0.7.100.2642 > 142.176.33.170.http: R 
3340982500:3340982500(0) ack 4135371285 win 0
15:19:24.001820 IP 10.0.5.13.2023 > 65.54.194.118.http: F 0:0(0) ack 1 
win 64790

It appears the offending traffic is not payload type traffic and is just 
control traffic, although our testing was only looking at a small dump 
with Ethereal.

Is there something we may have been setting up incorrectly?
JES

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[Francesco Ciocchetti]

James MacLean wrote:

> Hi Folks,
>
> Today we noticed that some traffic appears to be getting by netfilter 
> unNATed. The computers use typical SNATing such as :
>
> iptables -t nat -I POSTROUTING -j SNAT -s <private ip space> -o eth0 
> --to <IP of eth0>
>
a full , or almost full, iptables rules dump would be better.

> While we were dumping traffic on the public interface, we noticed 
> private IPs showing up in our dumps.
>
> We verified that there existed sessions between private IPs and public 
> Internet sites where some of the private traffic was returning to the 
> Internet site not NATed.
>
I hope that my "English knowledge' is joking me here ... how can be 
established session beetween you 'private IPs ' and Internet Server??
This is really impossible ... your packets, with private ip source, 
would never reach any internet server cause at least your ISP routers 
will surely filter on RFC 1918 addresses.

> We started watching, and saw this occur on servers with 2.6.11 and 
> 2.6.10 kernels which are all we have to test with.
>
> These kernels were compiled locally.
>
> For example, tcpdumping on a public interface where no private IPs 
> should live :
>
> 15:19:22.066190 IP 10.0.7.100.2640 > 142.176.33.171.http: R 
> 2946695981:2946695981(0) ack 4138631275 win 0
> 15:19:22.066516 IP 10.0.7.100.2641 > 142.176.33.170.http: R 
> 818314876:818314876(0) ack 4144245614 win 0
> 15:19:22.066684 IP 10.0.7.100.2638 > 142.176.33.170.http: R 
> 3381617258:3381617258(0) ack 4136231656 win 0
> 15:19:22.067353 IP 10.0.7.100.2639 > 142.176.33.170.http: R 
> 3920569986:3920569986(0) ack 4133397861 win 0
> 15:19:22.068045 IP 10.0.7.100.2642 > 142.176.33.170.http: R 
> 3340982500:3340982500(0) ack 4135371285 win 0
> 15:19:24.001820 IP 10.0.5.13.2023 > 65.54.194.118.http: F 0:0(0) ack 1 
> win 64790
>
>
this dump is really weird ... here we see packets from your private IP 
to public IP with RST and FIN flags ... how can you rst or fin a 
connection if you never established it? i would expect some SYN that 
never would get any ACK ... but RST and FIN are really really weird.

are you sure you have run tcpdump on public interface?

> Is there something we may have been setting up incorrectly?
> JES

i really think so :)

bye
P.

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[James MacLean]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

Francesco Ciocchetti wrote:

> James MacLean wrote:
>
>> While we were dumping traffic on the public interface, we noticed 
>> private IPs showing up in our dumps.
>>
>> We verified that there existed sessions between private IPs and 
>> public Internet sites where some of the private traffic was returning 
>> to the Internet site not NATed.
>>
> I hope that my "English knowledge' is joking me here ... how can be 
> established session beetween you 'private IPs ' and Internet Server??
> This is really impossible ... your packets, with private ip source, 
> would never reach any internet server cause at least your ISP routers 
> will surely filter on RFC 1918 addresses.
>
There is a session properly NATed between the client and the server. It
just happens that some traffic in that session escapes from being NATed
on it's way from the client (private IP space) to the server. How far it
gets we do not know :).

>> We started watching, and saw this occur on servers with 2.6.11 and 
>> 2.6.10 kernels which are all we have to test with.
>>
>> These kernels were compiled locally.
>>
>> For example, tcpdumping on a public interface where no private IPs 
>> should live :
>>
>> 15:19:22.066190 IP 10.0.7.100.2640 > 142.176.33.171.http: R 
>> 2946695981:2946695981(0) ack 4138631275 win 0
>> 15:19:22.066516 IP 10.0.7.100.2641 > 142.176.33.170.http: R 
>> 818314876:818314876(0) ack 4144245614 win 0
>> 15:19:22.066684 IP 10.0.7.100.2638 > 142.176.33.170.http: R 
>> 3381617258:3381617258(0) ack 4136231656 win 0
>> 15:19:22.067353 IP 10.0.7.100.2639 > 142.176.33.170.http: R 
>> 3920569986:3920569986(0) ack 4133397861 win 0
>> 15:19:22.068045 IP 10.0.7.100.2642 > 142.176.33.170.http: R 
>> 3340982500:3340982500(0) ack 4135371285 win 0
>> 15:19:24.001820 IP 10.0.5.13.2023 > 65.54.194.118.http: F 0:0(0) ack 
>> 1 win 64790
>>
>>
> this dump is really weird ... here we see packets from your private IP 
> to public IP with RST and FIN flags ... how can you rst or fin a 
> connection if you never established it? i would expect some SYN that 
> never would get any ACK ... but RST and FIN are really really weird.
>
This dump is only part of the traffic between the two nodes. The other
properly NATed traffic was left out.

> are you sure you have run tcpdump on public interface?
>
Definitely. We have tried now from 3 boxen at 2 different sites with the
same result.

>> Is there something we may have been setting up incorrectly?
>> JES
>
> i really think so :)
>
> bye
> P.

Us too ;).

Could not give an iptables dump from this site as it is huge and
revealing, but will see if the table from the second site is available.

Any other questions or comments welcome,
JES

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[James MacLean]

James MacLean wrote:

>
> Could not give an iptables dump from this site as it is huge and
> revealing, but will see if the table from the second site is available.
>
Here is the table from the second site:

[#] iptables -L -nv -t nat
Chain PREROUTING (policy ACCEPT 825K packets, 67M bytes)
 pkts bytes target     prot opt in     out     source destination
 1156 55754 REDIRECT   tcp  --  eth1   *       0.0.0.0/0 
0.0.0.0/0           multiport dports 21,2222,3210 redir ports 2370
20841 1015K ACCEPT     tcp  --  eth1   *       0.0.0.0/0 
142.177.0.0/16      tcp dpt:80
 193K 9354K REDIRECT   tcp  --  eth1   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:80 redir ports 3128

Chain POSTROUTING (policy ACCEPT 882K packets, 54M bytes)
 pkts bytes target     prot opt in     out     source destination
80991 3767K ACCEPT     all  --  *      *       142.177.178.40 0.0.0.0/0
   24  3796 ACCEPT     all  --  *      *       142.177.178.15 0.0.0.0/0
 428K   26M SNAT       all  --  *      eth0   !142.177.178.60 
0.0.0.0/0           to:142.227.188.60

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source destination

JES

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[James MacLean]

[-- Attachment #1: Type: text/plain, Size: 4558 bytes --]

Francesco Ciocchetti wrote:

> James MacLean wrote:
>
>> There is a session properly NATed between the client and the server. 
>> It just happens that some traffic in that session escapes from being 
>> NATed on it's way from the client (private IP space) to the server. 
>> How far it gets we do not know :).
>
>
> the problem is that some traffic can't excape from session :)
> I mean, the NAT table in netfilter is match only once per session. The 
> first packet is SNATted or DNATted , then all sequent packets of the 
> same connection does not touch the nat table , instead it is NATTED 
> based on connection tables.

Understood. And please excuse my lack of technical knowledge about the 
whole ip_conntrack experience :). We here are long time users and have 
never seen this situation before. Especially on multiple boxes. So I 
thought it was best to get the info out to this list in case there is 
more to it then bad configuration on our part ;(. If it was only 
occurring on one box, we'd kick it around more, but it appears to be a 
little more pronounced ;).

> I have a 2.6.11 firewall , that was a 2.6.10 before, and i never had 
> such an experience ... my packets never escaped from session table :)
>
Ok. But then explain how _any_ traffic can get through unNATed via a 
table where all private IPs are SNATed according to the rules. This is 
what we can not understand. Even adding an SNAT to the very top of the 
POSTROUTING chain didn't help.

We only tripped over this by accident. Otherwise, we'd still not have 
any clue that this was happening.

> I would try to flush iptables rules and try with just a few rules with 
> NAT to check if it does work or not ... then insert your rules by step 
> until you find where your problems lives.

Appreciate your thoughts. These sites are a bit to busy to probably do 
it that way, but we'll see if we can get a controlled environment to 
test in.

Some more dump, just for fun :)...

08:30:11.100766 IP 10.0.5.243.1270 > 63.250.195.12.http: F 
3707121597:3707121597(0) ack 502138606 win 64970
08:30:17.054926 IP 142.177.51.51.36703 > 63.250.195.12.http: S 
978075472:978075472(0) win 5840 <mss 1460,sackOK,timestamp 1145498148 
0,nop,wscale 2>
08:30:17.136969 IP 63.250.195.12.http > 142.177.51.51.36703: S 
1839730991:1839730991(0) ack 978075473 win 65535 <mss 1460,nop,wscale 
1,nop,nop,timestamp 1702037157 1145498148>
08:30:17.136998 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 1 win 
1460 <nop,nop,timestamp 1145498230 1702037157>
08:30:17.137710 IP 142.177.51.51.36703 > 63.250.195.12.http: P 
1:552(551) ack 1 win 1460 <nop,nop,timestamp 1145498230 1702037157>
08:30:17.221654 IP 63.250.195.12.http > 142.177.51.51.36703: F 
521:521(0) ack 552 win 33304 <nop,nop,timestamp 1702037165 1145498230>
08:30:17.221695 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 1 win 
1460 <nop,nop,timestamp 1145498314 1702037157>
08:30:17.221777 IP 63.250.195.12.http > 142.177.51.51.36703: P 
1:521(520) ack 552 win 33304 <nop,nop,timestamp 1702037165 1145498230>
08:30:17.221799 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 522 
win 1728 <nop,nop,timestamp 1145498315 1702037165>
08:30:17.222489 IP 142.177.51.51.36703 > 63.250.195.12.http: F 
552:552(0) ack 522 win 1728 <nop,nop,timestamp 1145498315 1702037165>
08:30:17.304268 IP 63.250.195.12.http > 142.177.51.51.36703: . ack 553 
win 33304 <nop,nop,timestamp 1702037174 1145498315>
08:30:32.472154 IP 10.0.4.10.2107 > 63.250.195.12.http: F 
3949942874:3949942874(0) ack 724796989 win 63967
08:30:34.730636 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1 
win 63967
08:30:39.345472 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1 
win 63967
08:30:48.577198 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1 
win 63967
08:31:07.039327 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1 
win 63967
08:31:43.960191 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1 
win 63967

This traffic happens to include both interfaces, but the last spurt from 
10.0.4.10 shows up on the public side of the interface. This box runs 
squid incase you were wondering why there was not a direct handoff per 
packet, but other boxes with this problem are not running squid.

May I suggest someone else even try it at home :), or on a half busy 
box? We _are_ honestly seeing this at different sites with different 
rules, but with the common SNAT for private IP space.

We are just dumping traffic to watch using tcpdump: tcpdump -i <public 
interface> -n net 10.0.0.0/8 or net 192.168.0.0/16

thanks again,
JES

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11tested

[Clist]

i have kernel 2.6.9

and that's true i see packets form internal network 192.168.0.0/16 appearing 
on externel iface besides all network should be natted form a range of 
192.168.x.y to 212.128.a.b.

It seems some packets missed the nat rule on iptables or connection tracking 
doesnt catch som packets when my box is busy.

Anyone got this behavior?



El Miércoles 16 Marzo 2005 13:49, James MacLean escribió:
> Francesco Ciocchetti wrote:
> > James MacLean wrote:
> >> There is a session properly NATed between the client and the server.
> >> It just happens that some traffic in that session escapes from being
> >> NATed on it's way from the client (private IP space) to the server.
> >> How far it gets we do not know :).
> >
> > the problem is that some traffic can't excape from session :)
> > I mean, the NAT table in netfilter is match only once per session. The
> > first packet is SNATted or DNATted , then all sequent packets of the
> > same connection does not touch the nat table , instead it is NATTED
> > based on connection tables.
>
> Understood. And please excuse my lack of technical knowledge about the
> whole ip_conntrack experience :). We here are long time users and have
> never seen this situation before. Especially on multiple boxes. So I
> thought it was best to get the info out to this list in case there is
> more to it then bad configuration on our part ;(. If it was only
> occurring on one box, we'd kick it around more, but it appears to be a
> little more pronounced ;).
>
> > I have a 2.6.11 firewall , that was a 2.6.10 before, and i never had
> > such an experience ... my packets never escaped from session table :)
>
> Ok. But then explain how _any_ traffic can get through unNATed via a
> table where all private IPs are SNATed according to the rules. This is
> what we can not understand. Even adding an SNAT to the very top of the
> POSTROUTING chain didn't help.
>
> We only tripped over this by accident. Otherwise, we'd still not have
> any clue that this was happening.
>
> > I would try to flush iptables rules and try with just a few rules with
> > NAT to check if it does work or not ... then insert your rules by step
> > until you find where your problems lives.
>
> Appreciate your thoughts. These sites are a bit to busy to probably do
> it that way, but we'll see if we can get a controlled environment to
> test in.
>
> Some more dump, just for fun :)...
>
> 08:30:11.100766 IP 10.0.5.243.1270 > 63.250.195.12.http: F
> 3707121597:3707121597(0) ack 502138606 win 64970
> 08:30:17.054926 IP 142.177.51.51.36703 > 63.250.195.12.http: S
> 978075472:978075472(0) win 5840 <mss 1460,sackOK,timestamp 1145498148
> 0,nop,wscale 2>
> 08:30:17.136969 IP 63.250.195.12.http > 142.177.51.51.36703: S
> 1839730991:1839730991(0) ack 978075473 win 65535 <mss 1460,nop,wscale
> 1,nop,nop,timestamp 1702037157 1145498148>
> 08:30:17.136998 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 1 win
> 1460 <nop,nop,timestamp 1145498230 1702037157>
> 08:30:17.137710 IP 142.177.51.51.36703 > 63.250.195.12.http: P
> 1:552(551) ack 1 win 1460 <nop,nop,timestamp 1145498230 1702037157>
> 08:30:17.221654 IP 63.250.195.12.http > 142.177.51.51.36703: F
> 521:521(0) ack 552 win 33304 <nop,nop,timestamp 1702037165 1145498230>
> 08:30:17.221695 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 1 win
> 1460 <nop,nop,timestamp 1145498314 1702037157>
> 08:30:17.221777 IP 63.250.195.12.http > 142.177.51.51.36703: P
> 1:521(520) ack 552 win 33304 <nop,nop,timestamp 1702037165 1145498230>
> 08:30:17.221799 IP 142.177.51.51.36703 > 63.250.195.12.http: . ack 522
> win 1728 <nop,nop,timestamp 1145498315 1702037165>
> 08:30:17.222489 IP 142.177.51.51.36703 > 63.250.195.12.http: F
> 552:552(0) ack 522 win 1728 <nop,nop,timestamp 1145498315 1702037165>
> 08:30:17.304268 IP 63.250.195.12.http > 142.177.51.51.36703: . ack 553
> win 33304 <nop,nop,timestamp 1702037174 1145498315>
> 08:30:32.472154 IP 10.0.4.10.2107 > 63.250.195.12.http: F
> 3949942874:3949942874(0) ack 724796989 win 63967
> 08:30:34.730636 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1
> win 63967
> 08:30:39.345472 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1
> win 63967
> 08:30:48.577198 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1
> win 63967
> 08:31:07.039327 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1
> win 63967
> 08:31:43.960191 IP 10.0.4.10.2107 > 63.250.195.12.http: F 0:0(0) ack 1
> win 63967
>
> This traffic happens to include both interfaces, but the last spurt from
> 10.0.4.10 shows up on the public side of the interface. This box runs
> squid incase you were wondering why there was not a direct handoff per
> packet, but other boxes with this problem are not running squid.
>
> May I suggest someone else even try it at home :), or on a half busy
> box? We _are_ honestly seeing this at different sites with different
> rules, but with the common SNAT for private IP space.
>
> We are just dumping traffic to watch using tcpdump: tcpdump -i <public
> interface> -n net 10.0.0.0/8 or net 192.168.0.0/16
>
> thanks again,
> JES

-- 
-------------------------------------------------
Clister UAH
-------------------------------------------------

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[Marius Mertens]

On Wednesday, March 16, 2005 1:49 PM,
James MacLean wrote:

> [...]
> May I suggest someone else even try it at home :), or on a half busy
> box? We _are_ honestly seeing this at different sites with different
> rules, but with the common SNAT for private IP space.
> [...]

Sorry I cannot provide anything to solve your problem, but maybe you want to 
check the following:
I also had (and already have, I just ignore it at the moment) a quite 
similar problem: Some packets that should have been modified by NAT were not 
processed, but in the direction "Internet --> NATted Clients" (exactly the 
opposite direction that makes problems on your setup) so that missed packets 
hit the INPUT rules of my router.
If you want to have more detailed information please see 
http://lists.netfilter.org/pipermail/netfilter/2005-January/057795.html
Now to the property you might want to check: All packets being not correctly 
processed by NAT had the state INVALID. I am not sure when/why the 
connection became INVALID, but since there has been traffic in both 
directions before, it it unlikely that it was INVALID in the first place.
Perhaps your not processed packets are also considered INVALID?
This is of course far away from a solution (since it is still unclear, why 
they become INVALID), but if we can find further criteria that applies to 
all these similar problems, maybe we are able to track it down.

Marius 

Re: Private traffic seen on public NATed interface - Linux 2.6.10-11 tested

[James MacLean]

[-- Attachment #1.1: Type: text/plain, Size: 2137 bytes --]

Marius Mertens wrote:

> On Wednesday, March 16, 2005 1:49 PM,
> James MacLean wrote:
>
>> [...]
>> May I suggest someone else even try it at home :), or on a half busy
>> box? We _are_ honestly seeing this at different sites with different
>> rules, but with the common SNAT for private IP space.
>> [...]
>
>
> Sorry I cannot provide anything to solve your problem, but maybe you 
> want to check the following:
> I also had (and already have, I just ignore it at the moment) a quite 
> similar problem: Some packets that should have been modified by NAT 
> were not processed, but in the direction "Internet --> NATted Clients" 
> (exactly the opposite direction that makes problems on your setup) so 
> that missed packets hit the INPUT rules of my router.
> If you want to have more detailed information please see 
> http://lists.netfilter.org/pipermail/netfilter/2005-January/057795.html
> Now to the property you might want to check: All packets being not 
> correctly processed by NAT had the state INVALID. I am not sure 
> when/why the connection became INVALID, but since there has been 
> traffic in both directions before, it it unlikely that it was INVALID 
> in the first place.
> Perhaps your not processed packets are also considered INVALID?
> This is of course far away from a solution (since it is still unclear, 
> why they become INVALID), but if we can find further criteria that 
> applies to all these similar problems, maybe we are able to track it 
> down.
>
> Marius
>
Bingo. And thanks :). Yes, this is looking very similar to our 
situation. A small dump and the matching INVALID rule logging :

12:24:59.083117 IP 10.0.5.221.1672 > 64.202.98.35.http: F 
1992443149:1992443149(0) ack 2731371818 win 63513

Mar 16 12:24:5 the kernel: INVALID IN=eth1 OUT=eth0 SRC=10.0.5.221 
DST=64.202.98.35 LEN=40 TOS=0x00 PREC=0x00 TTL=127 ID=15881 DF PROTO=TCP 
SPT=1672 DPT=80 WINDOW=63513 RES=0x00 ACK FIN URGP=0

Watching the logs on busy sites we see many of these :).

So now we know what it is, and we can simply apply INVALID rules if we 
need to. I wonder how long this has been going on :(.

thanks again,
JES


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3684 bytes --]