NAT Port Forward problem in a not so simple network

NAT Port Forward problem in a not so simple network

[Fabio De Paolis]

Hi to all,

I'm sure this thread is quite old,
but I would like to ask to this nice mailing list what is best way to
achieve a bandwidth limit with IpTables/NetFilter

I have several server with Debian Sarge stable
Linux 2.6.18-6-686 #1 SMP i686 GNU/Linux

And with IpTables v1.3.6

The bandwidth of the network is 10Mbit
I would like to limit the network traffic for each server to 250KByte/sec
This limit should be for upstram and dowstream

I searched around and have read a lot about:
shaper, wondershaper, tc, qdisc, etc.
And also about monitoring to see if it work:
ntop, sntop, iptraf, etc.

But I'm not sure what are pro and cons of each method.
I'm sure you can tell more!

Thanks in advance.

Re: NAT Port Forward problem in a not so simple network

[Marek Kierdelewicz]

>Hi to all,

Hi,

>This limit should be for upstram and dowstream
>I searched around and have read a lot about:
>shaper, wondershaper, tc, qdisc, etc.
>But I'm not sure what are pro and cons of each method.
>I'm sure you can tell more!

I recommend using tc tool[1]& IFB device[2] & htb qdisc[3] & u32
filter[4] for the job. Following page contains HTB user guide with
examples[5]. Building configuration to suit your needs shouldn't be too
hard. If you have any problems just ask.

General idea:
- Let's assume eth0 is your upstream interface (to Internet) that you do
NAT on and eth1 is downstream interface connected to servers,
- Downstream shaping would be done on eth1,
- Ingress traffic on eth1 would be redirected to ifb0 [2],
- Upstream shaping would be done on ifb0 (source addresses of
traffic from your servers would be pre-nat addresses, so no need for
iptables packet marking - u32 filter[4] is usable).

>And also about monitoring to see if it work:
>ntop, sntop, iptraf, etc.

tc tool[1] with specific parameters (eg. "tc -s -d class sh dev eth0")
can show you rates and pps of each class on specified interface.

Good luck in your endeavours.

[1] http://linux.die.net/man/8/tc
[2] http://www.linuxfoundation.org/en/Net:IFB
[3] http://linux.die.net/man/8/tc-htb
[4] http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.adv-filter.u32.html
[5] http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

Regards,
Marek Kierdelewicz

NAT Port Forward problem in a not so simple network

[Fabio De Paolis]

Hello to all,

I have a NAT/Forward problem and I'm unable to solve, any help will be 
greatly appreciated.

I have a service that runs on a server that has no public IP, another 
server with the public IP should forward the port to it.

Scenario:

/--------\
|A:Cisco1|
\--------/
   .
   ............   
   .          .
/--------\  /--------\
|B:Cisco2|  |C:HPpro1|
\--------/  \--------/
   .
   .
/--------\
|D:Works1|
\--------/

Cisco1: Router where I can change nothing
Cisco2: Router where I can make very little changes
HPpro1: Server with Linux (Debian) and IPTables where I can change all 
*HELP HERE*
Works1: Workstation with a service listening on port 8080 TCP

A: Cisco1:
It has several public IPs and it redirect 1 IP on HPpro (all ports and 
protocols)
LAN interface has 192.168.0.1, all works well here.
It has a port forward cisco rule that works (tested) it is:
ip nat inside source static 192.168.0.11 $PUBLICIP extendable (it is 
like NETMAP)

B: Cisco2:
It is used to let Workstation surf on the net
WAN interface has 192.168.0.2
LAN interface has 192.168.1.1, workstation are 192.168.1.2-20
It has a port forward cisco rule that works (tested) it is:
ip nat inside source static tcp 192.168.1.2 8080 192.168.0.2 8080 extendable

C: HPpro1: *HELP HERE*
It is a production server with many services on it
I would like to use its IP to test some application on a workstation 
instead of the production server
ETH0: 192.168.0.11
I need that it forward every packet that comes on 192.168.0.11:8080 
(from 192.168.0.1) to 192.168.0.2

D: Works1
It has a service listening on port 8080
ETH0: 192.168.1.2


Lets say that everything works fine if I try to connect to
192.168.0.2:8080 (from 192.168.0.6 for example) I access to 
192.168.1.2:8080 without problem

I need to access even connecting to 192.168.0.11:8080 (that is the IP 
with where if forwarded the public IP)


My try was this:
I create a simple port forward like this
DNAT tcp -- 0.0.0.0/0 192.168.0.11 tcp spts:1024:65535 dpt:41080 
flags:0x17/0x02 state NEW to:192.168.0.2

It does not work, I sniffed all the packets on the hub/switch and I see 
that:
#, Source IP, Source MAC, Destination IP, Destionation MAC, Protocol, Info
1, 192.168.0.1, Cisco1, 192.168.0.11, HPpro1, TCP, 1234 > 8080 [SYN] 
Seq=0 Len=0 MSS=1460
2, 192.168.0.1, HPpro1, 192.168.0.02, Cisco2, TCP, 1234 > 8080 [SYN] 
Seq=0 Len=0 MSS=1460
3, 192.168.0.2, Cisco2, 192.168.0.01, Cisco1, TCP, 8080 > 1234 [SYN, 
ACK] Seq=780620 Ack=1 Len=0 MSS=1460
4, 192.168.0.1, Cisco1, 192.168.0.02, Cisco2, TCP, 1234 > 8080 [RST] 
Seq=1 Len=0

Packet 1 is the SYN request from Cisco1 to HPpro1
Packet 2 is strange, HPpro1 forwards to Cisco2 using the IP of Cisco1
Packet 3 Cisco2 does his job and the give the reply to Cisco1
Packet 4 Cisco1 reset the communication, perhaps because he wait a reply 
from HPpro1 and not Cisco2?


Sorry for long message but it was to better explain the problem

Many thanks for you help

Re: NAT Port Forward problem in a not so simple network

[whiplash]

Fabio De Paolis ha scritto:

> Packet 4 Cisco1 reset the communication, perhaps because he wait a reply 
> from HPpro1 and not Cisco2?

Of course.
A possible solution is to SNAT packets exiting from 192.168.0.11
with destination 192.168.0.2.

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/15/08 07:15, whiplash wrote:
> Of course. A possible solution is to SNAT packets exiting from 
> 192.168.0.11 with destination 192.168.0.2.

*nod*

However, SNATing will have the possibly undesirable side effect of 
making the traffic appear to be from the machine that is doing the SNATing.



Grant. . . .

Re: NAT Port Forward problem in a not so simple network

[Pascal Hambourg]

Hello,

Grant Taylor a écrit :
> On 04/15/08 07:15, whiplash wrote:
> 
>> Of course. A possible solution is to SNAT packets exiting from 
>> 192.168.0.11 with destination 192.168.0.2.
> 
> *nod*
> 
> However, SNATing will have the possibly undesirable side effect of 
> making the traffic appear to be from the machine that is doing the SNATing.

It does not matter so much as, according to the packet trace, Cisco1 
appears to SNAT incoming connections forwarded to HPpro1 anyway. So all 
you get in the logs is HPpro1's address instead of Cisco1's...

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/17/08 09:49, Pascal Hambourg wrote:
> It does not matter so much as, according to the packet trace, Cisco1 
> appears to SNAT incoming connections forwarded to HPpro1 anyway. So all 
> you get in the logs is HPpro1's address instead of Cisco1's...

Huh!  Great!  Friggign lovely!

Do you know if that the prescribed Cisco way of doing that or just the 
way that it was configured at your site?

Seeing this new evidence, just DNAT & SNAT the traffic on the HP 
ProLiant and call it a day.



Grant. . . .

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/15/08 04:48, Fabio De Paolis wrote:
> I have a service that runs on a server that has no public IP, another 
> server with the public IP should forward the port to it.

Let's break this down in to the simple network flow before trying to 
make things work (namely to make sure I understand what you are wanting 
to do).

Let's do a re-draw with some clarifications.

        +---+
        | Z |
        +-+-+
          |
          :       (INet)
          |
        +-+-+
=======| A |============
        +-+-+
          |
      +---+---+   (DMZ)
      |       |
    +-+-+   +-+-+
===| B |===| C |========
    +-+-+   +---+
      |           (LAN)
    +-+-+
    | D |
    +---+

A = Cisco 1
B = Cisco 2
C = HP ProLiant 1
D = Workstation 1
Z = INet client

Note:  References to "External" (e) interfaces will be facing up and 
"Internal" (i) interfaces will be facing down.

I'm presuming (for the sake of discussion) that you presently you have 
globally routable services that appear to be on one or more Ae addresses 
that are port forwarded to Ce (and possibly others).

Now it sounds like you are wanting to port forward (one or more of) said 
service(s) from Ce to De with out changing any thing on A.  So the 
traffic flow would be from client Z to Ae to Ce to De and back in the 
same direction.

I'm also guessing that you would like all systems to have the capability 
(whether or not you use it or firewall it is up to you) to connect to Ce 
and have the connection be port forwarded to De.  Correct?

Before I go in to how to do this, please let me know if I'm on track or not.

> Many thanks for you help

*nod*



Grant. . . .

Re: NAT Port Forward problem in a not so simple network

[Fabio De Paolis]

Grant Taylor ha scritto:
> On 04/15/08 04:48, Fabio De Paolis wrote:
>> I have a service that runs on a server that has no public IP, another 
>> server with the public IP should forward the port to it.
>
> Let's break this down in to the simple network flow before trying to 
> make things work (namely to make sure I understand what you are 
> wanting to do).
>
> Let's do a re-draw with some clarifications.
>
>  Note:  References to "External" (e) interfaces will be facing up and 
> "Internal" (i) interfaces will be facing down.
>
> I'm presuming (for the sake of discussion) that you presently you have 
> globally routable services that appear to be on one or more Ae 
> addresses that are port forwarded to Ce (and possibly others).
>
> Now it sounds like you are wanting to port forward (one or more of) 
> said service(s) from Ce to De with out changing any thing on A.  So 
> the traffic flow would be from client Z to Ae to Ce to De and back in 
> the same direction.
>
> I'm also guessing that you would like all systems to have the 
> capability (whether or not you use it or firewall it is up to you) to 
> connect to Ce and have the connection be port forwarded to De.  Correct?
>
> Before I go in to how to do this, please let me know if I'm on track 
> or not.
>
>
Absoluttely CORRECT, your description is very very good.

Another goal should be to minimize traffic on C for service running on D.

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/15/08 11:22, Fabio De Paolis wrote:
> Absoluttely CORRECT, your description is very very good.

*nod*  Now I know that I am on track and that it is safe to go down the 
path that I was thinking about.

> Another goal should be to minimize traffic on C for service running on D.

Hum.  This new goal may be problematic.  The problem is that A is 
DNATing traffic to C that you now want to be re-directed elsewhere.  So 
with out re-configuring A, the traffic is going to continue to be DNATed 
to C.  What is better in the long run is to have A DNAT the traffic to B 
which will then DNAT the traffic in to D.

How much control do you have over B?

Can you request changes be made to A on your behalf?

I recently helped someone else on this list with a similar scenario. 
However in their scenario both C and D were directly connected to the 
internet via different providers and there was a VPN between C and D. 
The goal was to port forward connections originally to C over to D and 
have the replies go back through C and out to the original client.  We 
ended up getting things to work exactly as they needed to.  However all 
the traffic for the forwarded service was still passing through C on its 
way to D, which you are now wanting to avoid.



Grant. . . .

Re: NAT Port Forward problem in a not so simple network

[Fabio De Paolis]

Grant Taylor ha scritto:
> On 04/15/08 11:22, Fabio De Paolis wrote:
>> Absoluttely CORRECT, your description is very very good.
>
> *nod*  Now I know that I am on track and that it is safe to go down 
> the path that I was thinking about.
>
>> Another goal should be to minimize traffic on C for service running 
>> on D.
>
> Hum.  This new goal may be problematic.  The problem is that A is 
> DNATing traffic to C that you now want to be re-directed elsewhere.  
> So with out re-configuring A, the traffic is going to continue to be 
> DNATed to C.  What is better in the long run is to have A DNAT the 
> traffic to B which will then DNAT the traffic in to D.
>
> How much control do you have over B?
>
I have total control on B, even if fewer changes is good.

> Can you request changes be made to A on your behalf?
A is black box, it is from my service provider, I can change nothing, 
also request will be discarded at 99%

>
> I recently helped someone else on this list with a similar scenario. 
> However in their scenario both C and D were directly connected to the 
> internet via different providers and there was a VPN between C and D. 
> The goal was to port forward connections originally to C over to D and 
> have the replies go back through C and out to the original client.  We 
> ended up getting things to work exactly as they needed to.  However 
> all the traffic for the forwarded service was still passing through C 
> on its way to D, which you are now wanting to avoid.
>
Yes on my knowledge I know that it can't be done without doubling the 
traffic on the net. I was wondering if at yuor knowledge the was another 
way.
Of course if I could nat a port from A to B it would be easy and the 
traffic will me at minimum, but it cant be done.
I was wondering if there was a way to use C only for initial handshake 
and not for all packets, but it seems no.


Actually I'm with this iptables rules

iptables -nL -t nat

PREROUTING
DNAT       tcp  --  0.0.0.0/0            192.168.0.11        tcp 
spts:1024:65535 dpts:8080 flags:0x17/0x02 state NEW to:192.168.0.2

POSTROUTING
SNAT       tcp  --  0.0.0.0/0            192.168.0.2         tcp 
spts:1024:65535 dpts:8080 flags:0x17/0x02 state NEW to:192.168.0.11

It seems to work but this is the traffic I see on the net for a single 
packet
#, Source IP(Source MAC), Destination IP(Destionation MAC), Protocol, Info
1, 192.168.0.01(Cisco1), 192.168.0.11(HPpro1), TCP, 1234 > 8080 [SYN]
2, 192.168.0.11(HPpro1), 192.168.0.02(Cisco2), TCP, 1234 > 8080 [SYN]
3, 192.168.0.02(Cisco2), 192.168.0.11(Cisco2), TCP, 8080 > 1234 [SYN, ACK]
4, 192.168.0.11(HPpro1), 192.168.0.01(Cisco1), TCP, 8080 > 1234 [SYN, ACK]
and so on...


This is technically a Bounce. Let me know if this setup is correct, thanks!

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/16/08 08:54, Fabio De Paolis wrote:
> I have total control on B, even if fewer changes is good.

Understood.

> A is black box, it is from my service provider, I can change nothing, 
> also request will be discarded at 99%

*sigh*  I really dislike incompetent / uncooperative service providers. 
  You know that I had to ask.

> Yes on my knowledge I know that it can't be done without doubling the 
> traffic on the net. I was wondering if at yuor knowledge the was 
> another way. Of course if I could nat a port from A to B it would be 
> easy and the traffic will me at minimum, but it cant be done. I was 
> wondering if there was a way to use C only for initial handshake and 
> not for all packets, but it seems no.

Strictly speaking, on layer 3, no there is not any thing (that I am 
aware of) that can be done.  However if you are willing to go down to in 
between layers 2 and 3 or even down to layer 2 there might be something 
that can be done.

          +---+
          | Z |
          +-+-+
            |
            :       (INet)
            |
          +-+-+
=========| A |============
          +-+-+
            |
    +---+---+---+   (DMZ)
    |   |       |
    | +-+-+   +-+-+
===|=| B |===| C |========
    | +-+-+   +---+
    |   |           (LAN)
    | +-+-+
    +-+ D |
      +---+

I'm guessing that there are other services on C that prevent you from 
moving it's IP to B.  Correct?

I'm not sure how well this will work out (read:  I don't know how well 
the Cisco will play in this game...) but you might be able to establish 
some sort of tunnel based forwarding from C to D so that inbound 
requests pass through the tunnel and replies go directly from D back out 
via A to the client.

Let's say for the sake of discussion that you add a connection from D 
back in to the DMZ (as above) and have this interface configured to 
*NOT* respond to ARP requests.  If you do this, you could have the same 
IP bound to C as well as the new DMZ facing interface on D.  With this 
type of set up, you could tunnel traffic from C to D via B and have D 
reply directly back with out passing through B or C.

In short, this is using the IP Tunnel mode of Linux Virtual Server to 
turn C in to a director for the single node back end.  As such, your 
client Z would connect to Ae which is port forwarded to Ce which is 
tunnel to D which processes and replies to the client from the same IP 
as Ce.  This means that A will send traffic to the IP that is bound to 
Ce and get replies from the same IP only bound to D's DMZ interface. 
The only difference that A should see is a different MAC address as the 
source for the reply traffic.  However, if you spoof the MAC address, 
this will not be a problem.  If you do spoof the MAC address you will 
need to do something like GARP to make sure the DMZ switch does not 
""learn that the location of the shared MAC address is where D's DMZ 
interface is connected.

> Actually I'm with this iptables rules
> 
> iptables -nL -t nat
> 
> PREROUTING
> DNAT       tcp  --  0.0.0.0/0            192.168.0.11        tcp spts:1024:65535 dpts:8080 flags:0x17/0x02 state NEW to:192.168.0.2
> 
> POSTROUTING
> SNAT       tcp  --  0.0.0.0/0            192.168.0.2         tcp spts:1024:65535 dpts:8080 flags:0x17/0x02 state NEW to:192.168.0.11

*nod*

> It seems to work but this is the traffic I see on the net for a single packet
> #, Source IP(Source MAC), Destination IP(Destionation MAC), Protocol, Info
> 1, 192.168.0.01(Cisco1), 192.168.0.11(HPpro1), TCP, 1234 > 8080 [SYN]
> 2, 192.168.0.11(HPpro1), 192.168.0.02(Cisco2), TCP, 1234 > 8080 [SYN]
> 3, 192.168.0.02(Cisco2), 192.168.0.11(Cisco2), TCP, 8080 > 1234 [SYN, ACK]
> 4, 192.168.0.11(HPpro1), 192.168.0.01(Cisco1), TCP, 8080 > 1234 [SYN, ACK]
> and so on...

As you can see, this will get the traffic where you want it to go, but 
it is far from the original traffic.  Namely your source IP now looks 
like C, thus foiling any thing in logs on D.

> This is technically a Bounce. Let me know if this setup is correct, 
> thanks!

I had not thought of what this might be called, but you are correct in 
that it will work with caveats.

Chew on the LVS idea and let me know if you want to run with it.



Grant. . . .

Re: NAT Port Forward problem in a not so simple network

[Fabio De Paolis]

Grant Taylor ha scritto:
>
>> Yes on my knowledge I know that it can't be done without doubling the 
>> traffic on the net. I was wondering if at yuor knowledge the was 
>> another way. Of course if I could nat a port from A to B it would be 
>> easy and the traffic will me at minimum, but it cant be done. I was 
>> wondering if there was a way to use C only for initial handshake and 
>> not for all packets, but it seems no.
>
> Strictly speaking, on layer 3, no there is not any thing (that I am 
> aware of) that can be done.  However if you are willing to go down to 
> in between layers 2 and 3 or even down to layer 2 there might be 
> something that can be done.
>
>          +---+
>          | Z |
>          +-+-+
>            |
>            :       (INet)
>            |
>          +-+-+
> =========| A |============
>          +-+-+
>            |
>    +---+---+---+   (DMZ)
>    |   |       |
>    | +-+-+   +-+-+
> ===|=| B |===| C |========
>    | +-+-+   +---+
>    |   |           (LAN)
>    | +-+-+
>    +-+ D |
>      +---+
>
> I'm guessing that there are other services on C that prevent you from 
> moving it's IP to B.  Correct?
>
> I'm not sure how well this will work out (read:  I don't know how well 
> the Cisco will play in this game...) but you might be able to 
> establish some sort of tunnel based forwarding from C to D so that 
> inbound requests pass through the tunnel and replies go directly from 
> D back out via A to the client.
>
> Let's say for the sake of discussion that you add a connection from D 
> back in to the DMZ (as above) and have this interface configured to 
> *NOT* respond to ARP requests.  If you do this, you could have the 
> same IP bound to C as well as the new DMZ facing interface on D.  With 
> this type of set up, you could tunnel traffic from C to D via B and 
> have D reply directly back with out passing through B or C.
>
> In short, this is using the IP Tunnel mode of Linux Virtual Server to 
> turn C in to a director for the single node back end.  As such, your 
> client Z would connect to Ae which is port forwarded to Ce which is 
> tunnel to D which processes and replies to the client from the same IP 
> as Ce.  This means that A will send traffic to the IP that is bound to 
> Ce and get replies from the same IP only bound to D's DMZ interface. 
> The only difference that A should see is a different MAC address as 
> the source for the reply traffic.  However, if you spoof the MAC 
> address, this will not be a problem.  If you do spoof the MAC address 
> you will need to do something like GARP to make sure the DMZ switch 
> does not ""learn that the location of the shared MAC address is where 
> D's DMZ interface is connected.
>
Thank you for the lesson.

D is too far from C's switch *sad*
and I think I can't add another cable

Re: NAT Port Forward problem in a not so simple network

[Grant Taylor]

On 04/18/08 08:43, Fabio De Paolis wrote:
> Thank you for the lesson.

*nod*

> D is too far from C's switch *sad* and I think I can't add another 
> cable

*nod*

Well at least you were able to get things to work, all be it 
sub-optimal.  Working is better than not.



Grant. . . .