Weird NAT problem

Weird NAT problem

[Bernhard Rosenkraenzer]

Hi,
I've set up a basic masquerading NAT box to share a PPPoE connection between 
various boxes.

I've done the same thing the same way before without running into trouble - 
but this time, clients have no problem keeping an existing transfer going, 
but initiating connections seems to be troublesome.

The basic setup: The line comes in to my eth0; I'm using rp-pppoe 3.10 and ppp 
2.4.4 (with kernel mode PPPoE enabled) on Linux 2.6.27.2, iptables userland 
1.4.1.1 (also tried 1.4.2-rc1, same results).
eth0 doesn't have an IP address assigned (but ppp0 on top of eth0 does).

eth1 uses IP address 192.168.1.1/255.255.255.0, and connects to the other 
boxes through a switch.

I'm doing

echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 >/proc/sys/net/ipv4/conf/all/forwarding
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -A FORWARD -i ppp0 -o eth1 -m state --state ESTABLISHED,RELATED -j 
ACCEPT
iptables -A FORWARD -i eth1 -o ppp0 -j ACCEPT
iptables -t nat -F
iptables -t nat -A POSTROUTING -s 192.168.2.0/255.255.255.0 -o ppp0 -j 
MASQUERADE

There are several public IPs on the PPPoE connection, so I use

iptables -t nat -A PREROUTING -d XXX.XXX.XXX.XXX -j DNAT --to-destination 
192.168.2.2
iptables -t nat -A POSTROUTING -s 192.168.2.2 -j SNAT --to-source 
XXX.XXX.XXX.XXX

to make 192.168.2.2 reachable from the outside world (but the troubles start 
before doing that).


On 192.168.2.2, a lot of things work:

$ ping ftp.ch.kernel.org
PING ftp.ch.kernel.org (212.40.5.79): 56 data bytes
64 bytes from 212.40.5.79: icmp_seq=0 ttl=61 time=11.645 ms
[...]
$ traceroute ftp.ch.kernel.org
traceroute to ftp.ch.kernel.org (212.40.5.79), 64 hops max, 40 byte packets
 1  192.168.2.1 (192.168.2.1)  0.551 ms  0.258 ms  0.231 ms
 2  bas-flu-as-10-01-lb-1.vtxnet.net (212.254.136.1)  10.988 ms  11.018 ms  
10.728 ms
 3  bas-flu-cr-76-02-ge-2-3.vtxnet.net (212.147.63.190)  11.476 ms  11.367 ms  
11.479 ms
 4  ftp.datacomm.ch (212.40.5.79)  10.739 ms  11.558 ms  10.972 ms

BUT

$ curl -o kernel.tar.bz2 
http://ftp.ch.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.tar.bz2

hangs without getting any data.


There is no problem with the line itself, because I can use everything on the 
router just fine, and I can even scp huge files from the outside to the 
public IP of 192.168.2.2 without running into the slightest problems.

I've tried doing some bad stuff like

iptables -P FORWARD ACCEPT

to make sure my rules are ok, and it didn't change anything.

192.168.2.2 is an OSX box - an OS I happen to know next to nothing about 
(Linux purist here...), 192.168.2.3 behaves the same way as 192.168.2.2, and 
is a Linux box running on 192.168.2.2 inside Parallels Server.

Any idea what's going wrong there? Is there anything I'm overlooking with my 
routing setup, or do I need to look for the problem on the OSX side?

Thanks
Bernhard

Re: Weird NAT problem

[Grant Taylor]

On 11/1/2008 5:07 PM, Bernhard Rosenkraenzer wrote:
> I've set up a basic masquerading NAT box to share a PPPoE connection 
> between various boxes.

*nod*

> I've done the same thing the same way before without running into 
> trouble - but this time, clients have no problem keeping an existing 
> transfer going, but initiating connections seems to be troublesome.

Ok...

Is it any connection or just some specific destinations?

> There is no problem with the line itself, because I can use 
> everything on the router just fine, and I can even scp huge files 
> from the outside to the public IP of 192.168.2.2 without running into 
> the slightest problems.

Ok...

> Any idea what's going wrong there? Is there anything I'm overlooking 
> with my routing setup, or do I need to look for the problem on the 
> OSX side?

Re: Weird NAT problem

[Grant Taylor]

(Let's try this again with out fat fingering the /Send/ button too soon.)

> Any idea what's going wrong there? Is there anything I'm overlooking 
> with my routing setup, or do I need to look for the problem on the OSX 
> side?

The first thing that comes to mind is that you may be dealing with an 
MTU / MSS issue.  Some of your connections work and some never seem to 
start, which is a classic symptom of an MTU / MSS mis-match with the 
ICMP error being filtered out along the way.

Try playing with the TCPMSS match extension and target to clamp the MSS 
to some lower values.  Try something extremely low and then start 
raising it until you start to have failures again and then back it down 
a bit.

PPPoE, despite being nice for providers, has many different problems 
that simple static IPs or even DHCP does not have.



Grant. . . .

Re: Weird NAT problem

[Bernhard Rosenkraenzer]

On Saturday 01 November 2008 23:52:25 Grant Taylor wrote:
> The first thing that comes to mind is that you may be dealing with an
> MTU / MSS issue.  Some of your connections work and some never seem to
> start, which is a classic symptom of an MTU / MSS mis-match with the
> ICMP error being filtered out along the way.

Thanks!!!! That's exactly what it turned out to be.

It's working perfectly now.

Thanks again
Bernhard

Re: Weird NAT problem

[Grant Taylor]

On 11/1/2008 6:52 PM, Bernhard Rosenkraenzer wrote:
> Thanks!!!! That's exactly what it turned out to be.

*nod*

IMHO PPPoE is only nice from the providers point of view.  I really wish 
that we could go to MPLS rather than PPPoE, but I don't think that will 
happen any time soon.

PPPoE does not do any thing for MTU / MSS issues, where as MPLS will 
split large frames for transport and then splice them back together on 
the other end.

> It's working perfectly now.

Good!

Don't you love it when it's something that is simple to solve / work 
around, even if should *NOT* have to.  ;)

> Thanks again

You are welcome.  :)



Grant. . . .

Re: Weird NAT problem

[Pascal Hambourg]

Hello,

Grant Taylor a écrit :
> 
> The first thing that comes to mind is that you may be dealing with an 
> MTU / MSS issue.  Some of your connections work and some never seem to 
> start, which is a classic symptom of an MTU / MSS mis-match with the 
> ICMP error being filtered out along the way.

Or with the ICMP error not being generated at all, because of the MTU 
mismatch not occuring at an IP device. This is quite common with a DSL 
modem acting as transparent bridge : as it works only at the link layer, 
when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it 
cannot either forward it on the ethernet link, fragment it or send an 
ICMP error.

The ISP routers should be responsible for doing the fragmentation and 
sending ICMP errors. But unfortunately they don't always do it. 
Sometimes the ISP don't even know the customer uses PPPoE, for example 
when it uses a third-party DSL carrier which deals with the PPPoX stuff 
and delivers user sessions to the ISP in L2TP tunnels.

> Try playing with the TCPMSS match extension and target to clamp the MSS 
> to some lower values.

Be aware that this workaround only works for TCP, not for other IP 
protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 6in4...

Re: Weird NAT problem

[Grant Taylor]

On 11/2/2008 5:08 AM, Pascal Hambourg wrote:
> Or with the ICMP error not being generated at all, because of the MTU 
> mismatch not occuring at an IP device. This is quite common with a DSL 
> modem acting as transparent bridge : as it works only at the link layer, 
> when it receives a PPPoE frame bigger than 1500 bytes from the ISP, it 
> cannot either forward it on the ethernet link, fragment it or send an 
> ICMP error.

Interesting point.  I had not considered a layer 2 device having to do 
some ""conversion and it not being able to.  This just further 
exasperates the problem.

> The ISP routers should be responsible for doing the fragmentation and 
> sending ICMP errors. But unfortunately they don't always do it. 
> Sometimes the ISP don't even know the customer uses PPPoE, for example 
> when it uses a third-party DSL carrier which deals with the PPPoX stuff 
> and delivers user sessions to the ISP in L2TP tunnels.

Do you ever wonder where the out sourcing will stop?  It seems like 
despite the best intentions of outsourcing, problems always seem to come up.

> Be aware that this workaround only works for TCP, not for other IP 
> protocols such as UDP, ICMP (big pings), GRE, IPSec (AH, ESP), IPIP, 
> 6in4...

Very good points.  /Fortunately/ I've not run in to them yet but I can 
see how it would certainly be possible.



Grant. . . .

Re: Weird NAT problem

[Bernhard Rosenkraenzer]

On Saturday 01 November 2008 23:48:34 Grant Taylor wrote:
> > I've done the same thing the same way before without running into
> > trouble - but this time, clients have no problem keeping an existing
> > transfer going, but initiating connections seems to be troublesome.
>
> Ok...
>
> Is it any connection or just some specific destinations?

It's almost any connection -- and the few connections that do work don't seem 
to be related to the destination. It seems more related to transfer size:

$ scp bero@linux.cd:linux*tar.bz2 .
Password:
linux-2.6.27.2.tar.bz2		0%	0	0.0 KB/s	- stalled -

So, authentication works, and it definitely gets something back or it wouldn't 
know the filename -- but it stops and just sits there as soon as bigger 
amounts of data get transferred.

It can't be a performance problem on the router - this is a quadcore box that 
is (currently) almost idle, and has virtually no traffic on either ethernet 
card.

Regards
Bernhard

Re: Weird NAT problem

[Grant Taylor]

On 11/1/2008 5:59 PM, Bernhard Rosenkraenzer wrote:
> It's almost any connection -- and the few connections that do work 
> don't seem to be related to the destination. It seems more related to 
> transfer size:

*nod*

> $ scp bero@linux.cd:linux*tar.bz2 .
> Password:
> linux-2.6.27.2.tar.bz2		0%	0	0.0 KB/s	- stalled -

I bet that if you sniff traffic on each end of the connection you will 
find that each end is sending packets and waiting for the other end to 
reply.  I'll even go so far as to say that the "linux.cd" system (above) 
is even sending the reply packets in response to the last packet that it 
received from your client and that your client is *not* receiving said 
reply packet.

> So, authentication works, and it definitely gets something back or it 
> wouldn't know the filename -- but it stops and just sits there as 
> soon as bigger amounts of data get transferred.

Ayup.  This is sounding more and more like a path MTU / (TCP)MSS issue 
to me.

Like I said before (I don't know if you read the second half of my 
post.) try using the TCPMSS match extension / target and clamping the 
TCPMSS value.

> It can't be a performance problem on the router - this is a quadcore 
> box that is (currently) almost idle, and has virtually no traffic on 
> either ethernet card.

I /seriously/ /doubt/ that the box is over loaded.  Keep in mind that 
SMP has some of its own issues with IPTables.  Though I don't think that 
is your problem here.



Grant. . . .