Hanging outgoing connections while incoming are OK

Hanging outgoing connections while incoming are OK

[Guido Anzuoni]

I'm setting up a firewall (CentOS 5.3) for our internet connection.
We have multiple internal networks so the fw has 5 network adapter.
Internet connection is managed by the ISP with 2 cisco router in LB.
I have absolutely no problem performing DNAT for incoming connections but
I am facing a strange problem with outgoing ones.
Let simplify the configuration.
Public addresses assigned: PUB_IP_OUTGOING, PUB_IP_OUTGOING
There is a subnet connecting the fw with the routers:
fw address: 10.254.254.1/24
Router1: 10.254.254.3/24
Router2: 10.254.254.4/24
Virtual ip for route: 10.254.254.2/24

fw default gateway: 10.254.254.2
fw eth0: 10.254.254.1, PUB_IP_OUTGOING, PUB_IP_OUTGOING

Internal networks:
fw eth1: 192.168.16.1
fw eth2: 192.168.17.1


The main iptables rule for outgoing connections is:
-A POSTROUTING -o eth0 -j SNAT --to-source PUB_IP_OUTGOING

Please note that we have a secondary internet connection managed by a
router at 192.168.16.80
so I have an additional rule:
-A POSTROUTING -o eth1 -j SNAT --to-source 192.168.16.1

Now, I have really strange behaviour browsing the web from internal networks
(well, even ssh outgoing connection from a windows box with putty
- connection is established but I have lost packets).
With certain urls the browser hangs waiting for something.
I have found one of these urls (an image perfectly valid about 7k)
and doing a wget of the same url
it downloads about 6k and hangs waiting for the remaining part
(beware that other urls for the same webserver perfectly work).
If I replace the default gateway with 192.168.16.1 I never miss a beat.
I have turned off one cisco router but nothing changes
(so LB should not be a problem - but, maybe, I have to turn off the
modem too... - ).
I have inspected with tcpdump the flow of packets and I have seen that the
requests come in from ethxx with INTERNAL_IP->TRAGET_WEB_SERVER pair
of addresses and
is outputted on eth0 with PUB_IP_OUTGOING->TRAGET_WEB_SERVER
and when I switch default gateway
is outputted on eth0 with 192.168.16.1->TRAGET_WEB_SERVER

My doubt is about eth0 configuration where I bind multiple addresses,
an internal one and all the public assigned by the ISP.
Is it a correct setup ?
I can't find anything wrong since certains connections are OK
while other are failing during packet flow.
More, when I switch gateway everything is perfectly working, but there
is a difference.
In this case, the SNAT address is on the same network of the router
while in the normal case
the SNAT address is the public one
(obviously, the ISP routes with PUB_IP_OUTGOING destination to 10.254.254.1).

TIA
Guido

Re: Hanging outgoing connections while incoming are OK

[Andrew Beverley]

On Sat, 2012-01-28 at 09:39 +0100, Guido Anzuoni wrote:
...
> fw default gateway: 10.254.254.2
> fw eth0: 10.254.254.1, PUB_IP_OUTGOING, PUB_IP_OUTGOING
...
> My doubt is about eth0 configuration where I bind multiple addresses,
> an internal one and all the public assigned by the ISP.
> Is it a correct setup ?

Unless I'm misunderstanding something, this does seem like a strange set
up. Why not just have the one IP address on eth0? Do the Cisco routers
also have an external IP address? Are these 2 completely independent WAN
links? If so, how is traffic shared between them?

Sorry for all the questions, but I'm not entirely understanding your set
up and what you are trying to achieve.

Andy

Re: Hanging outgoing connections while incoming are OK

[Guido Anzuoni]

OMG, sorry.
The intended setup was
fw eth0: 10.254.254.1, PUB_IP_INCOMING, PUB_IP_OUTGOING

I have used 1 public ip for incoming connections and 1 ip to
"masquerade" outgoing ones.
I don't know if it is necessary to bind public ip to some NIC in order
to let netfilter NAT work properly.

Anyway, there is a little progress in the analysys.
Starting a ssh session from a linux box on the internal network, I can
see with tcpdump a certain amount of packets
flowing along the path from the linux box, the firewall up to the ssh server.
Then packet flow stops for 10-20 seconds until final exchange takes
place and I have the shell prompt.
It seems like some packets start going round and round before arriving
to destination.
In fact, if I do several ls -l connection hangs again.
The strange thing is that there is no way to setup a connection if the
client is putty on a windows client.

Guido

On Tue, Jan 31, 2012 at 6:37 PM, Andrew Beverley <andy@andybev.com> wrote:
> On Sat, 2012-01-28 at 09:39 +0100, Guido Anzuoni wrote:
> ...
>> fw default gateway: 10.254.254.2
>> fw eth0: 10.254.254.1, PUB_IP_OUTGOING, PUB_IP_OUTGOING
> ...
>> My doubt is about eth0 configuration where I bind multiple addresses,
>> an internal one and all the public assigned by the ISP.
>> Is it a correct setup ?
>
> Unless I'm misunderstanding something, this does seem like a strange set
> up. Why not just have the one IP address on eth0? Do the Cisco routers
> also have an external IP address? Are these 2 completely independent WAN
> links? If so, how is traffic shared between them?
>
> Sorry for all the questions, but I'm not entirely understanding your set
> up and what you are trying to achieve.
>
> Andy
>
>

Re: Hanging outgoing connections while incoming are OK

[Andrew Beverley]

On Wed, 2012-02-01 at 16:40 +0100, Guido Anzuoni wrote:
> On Tue, Jan 31, 2012 at 6:37 PM, Andrew Beverley <andy@andybev.com> wrote:
> > On Sat, 2012-01-28 at 09:39 +0100, Guido Anzuoni wrote:
> > ...
> >> fw default gateway: 10.254.254.2
> >> fw eth0: 10.254.254.1, PUB_IP_OUTGOING, PUB_IP_OUTGOING
> > ...
> >> My doubt is about eth0 configuration where I bind multiple addresses,
> >> an internal one and all the public assigned by the ISP.
> >> Is it a correct setup ?
> >
> > Unless I'm misunderstanding something, this does seem like a strange set
> > up. Why not just have the one IP address on eth0? Do the Cisco routers
> > also have an external IP address? Are these 2 completely independent WAN
> > links? If so, how is traffic shared between them?
> >
> > Sorry for all the questions, but I'm not entirely understanding your set
> > up and what you are trying to achieve.

[please don't top-post]

> The intended setup was
> fw eth0: 10.254.254.1, PUB_IP_INCOMING, PUB_IP_OUTGOING
> 
> I have used 1 public ip for incoming connections and 1 ip to
> "masquerade" outgoing ones.
> I don't know if it is necessary to bind public ip to some NIC in order
> to let netfilter NAT work properly.

I'm still confused as to what you are trying to achieve with this set
up, and why you have several IP addresses eth0. I would suggest that
this is the cause of your problems.

> It seems like some packets start going round and round before arriving
> to destination.

Which would explain why you are seeing this.

Why not just have one public IP address on eth0 only? And then use that
for incoming and outgoing connections, at least until you've got it
working?

Something like:


[Internal
 interfaces]   __________
--------------|          |eth0
              |  Linux   |(PUB_IP_INCOMING only)
--------------|   Box    |------------------------->ISP Cisco router
              |          |      ^
--------------|__________|      |
                                |
                               SNAT here to PUB_IP_INCOMING

Andy

Re: Hanging outgoing connections while incoming are OK

[Guido Anzuoni]

>
> Which would explain why you are seeing this.
>
> Why not just have one public IP address on eth0 only? And then use that
> for incoming and outgoing connections, at least until you've got it
> working?
>
> Something like:
>
>
> [Internal
>  interfaces]   __________
> --------------|          |eth0
>              |  Linux   |(PUB_IP_INCOMING only)
> --------------|   Box    |------------------------->ISP Cisco router
>              |          |      ^
> --------------|__________|      |
>                                |
>                               SNAT here to PUB_IP_INCOMING
>
> Andy
>
>

First, sorry for the top-post, I just start typing without thinking.....
Yesterday I have tested your setup simply assigning one public address
( I have 6 addresses) to a machine on the network
and another to the router but the problem is still there.
I think that ISP should monitor the external interfaces of both router
to check if and when packets come in.
Anyway, I don't want to bother netfilter users with connection
problems that are out of scope.
The much more specific question is: in order to correctly perform SNAT
and DNAT, is it necessary to bind the referenced addresses
to some interface ?
I think the answer is no, it is not necessary, but I would like to
have a confirmation on that.

Guido

Re: Hanging outgoing connections while incoming are OK

[Andrew Beverley]

On Thu, 2012-02-02 at 08:48 +0100, Guido Anzuoni wrote:
> The much more specific question is: in order to correctly perform SNAT
> and DNAT, is it necessary to bind the referenced addresses
> to some interface ?

Well, I can't see why you'd want to SNAT to an IP address that isn't
assigned to the interface in question? I don't know whether it would
work or not though.

As for DNAT, the destination address wouldn't necessarily be on the same
machine, so the answer is no.

Unless I'm misunderstanding your question?

Andy