can't port forward on multihome

can't port forward on multihome

[sean darcy]

I have a multihomed server: eth0 is a static T1, and eth3 is a Verizon 
dsl line. I want eth3 as the default for general traffic, and eth0 for 
VOIP traffic.

eth1 is the internal interface. eth3 works fine as the masquerade out 
for NAT'd lan.

I've used ip to set up eth0 so I can ssh into it:

## eth0 is static
ETH0_IP_ADDR=www.xxx.yyy.zzz
ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
## this is the route through the gateway ip
ip route add default via <eth0 gateway ip> table 128


and that works. Which is important since that's the static address; the 
Verizon dsl address is dynamic.

The VOIP server ( asterisk ) is on the lan. I've tried to port forward 
ssh to the voip server:

$IPT -t nat -A PREROUTING  -p tcp --dport 2280 -j DNAT --to 10.10.10.180:22
$IPT -A FORWARD -p tcp --dport 22 -m state --state NEW -d 10.10.10.180 
-j ACCEPT

This works if I ssh to the eth3, the dynamic dsl interface:

ssh -p 2280 voip@<dsl ip address>

I get an ssh session on the voip server.

But:

ssh -p 2280 voip@<static ip address>

doesn't work. But I need to have others access the voip server using a 
static ip, but not give them access to the multihomed server.

AFAICT, the ssh on the voip server never sees anything ( at LogLevel 
DEBUG1 ) if I try over the static interface.

I assume I need some additional ip magic, but I'm clueless as to what's 
needed.

Thanks for any help.

sean

Re: can't port forward on multihome

[Pascal Hambourg]

Hello,

sean darcy a écrit :
> I have a multihomed server: eth0 is a static T1, and eth3 is a Verizon 
> dsl line. I want eth3 as the default for general traffic, and eth0 for 
> VOIP traffic.
> 
> eth1 is the internal interface. eth3 works fine as the masquerade out 
> for NAT'd lan.
> 
> I've used ip to set up eth0 so I can ssh into it:
> 
> ## eth0 is static
> ETH0_IP_ADDR=www.xxx.yyy.zzz
> ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
> ## this is the route through the gateway ip
> ip route add default via <eth0 gateway ip> table 128
> 
> and that works. Which is important since that's the static address; the 
> Verizon dsl address is dynamic.
> 
> The VOIP server ( asterisk ) is on the lan. I've tried to port forward 
> ssh to the voip server:
> 
> $IPT -t nat -A PREROUTING  -p tcp --dport 2280 -j DNAT --to 10.10.10.180:22
> $IPT -A FORWARD -p tcp --dport 22 -m state --state NEW -d 10.10.10.180 
> -j ACCEPT
> 
> This works if I ssh to the eth3, the dynamic dsl interface:
> 
> ssh -p 2280 voip@<dsl ip address>
> 
> I get an ssh session on the voip server.
> 
> But:
> 
> ssh -p 2280 voip@<static ip address>
> 
> doesn't work. But I need to have others access the voip server using a 
> static ip, but not give them access to the multihomed server.


The ip rule won't work for reply packets sent by the server, because . 
source address mangling occurs after the routing decision so the source 
address is 10.10.10.180, not (yet) eth0's address. If Verizon drops 
packets sent with a source address other than the one assigned to eth3, 
then the client won't receive any reply and the connection will fail.

In order to route the reply packets using table 128, you need to 
identify them. I guess that 10.10.10.180:22 as the source address:port 
is not discriminant enough, as it matches connections forwarded from 
eth3 too.

You can use the CONNMARK target to mark the incoming connection on eth0 
and copy the connection mark to the reply packets on eth1. Then you can 
use the packet mark in an ip rule.

iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
   -j CONNMARK --set-mark 0x1
iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark
ip rule add fwmark 0x1 table 128 prio 127

As you used DNAT, you may use the --ctorigdst option of the 'conntrack' 
match and mark reply packets based on the original destination address 
of the connection.

iptables -t mangle -A PREROUTING -i eth1 \
   -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
ip rule add fwmark 0x1 table 128 prio 127

Re: can't port forward on multihome

[sean darcy]

Pascal Hambourg wrote:
> Hello,
> 
> sean darcy a écrit :
>> I have a multihomed server: eth0 is a static T1, and eth3 is a Verizon 
>> dsl line. I want eth3 as the default for general traffic, and eth0 for 
>> VOIP traffic.
>>
>> eth1 is the internal interface. eth3 works fine as the masquerade out 
>> for NAT'd lan.
>>
>> I've used ip to set up eth0 so I can ssh into it:
>>
>> ## eth0 is static
>> ETH0_IP_ADDR=www.xxx.yyy.zzz
>> ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
>> ## this is the route through the gateway ip
>> ip route add default via <eth0 gateway ip> table 128
>>
>> and that works. Which is important since that's the static address; 
>> the Verizon dsl address is dynamic.
>>
>> The VOIP server ( asterisk ) is on the lan. I've tried to port forward 
>> ssh to the voip server:
>>
>> $IPT -t nat -A PREROUTING  -p tcp --dport 2280 -j DNAT --to 
>> 10.10.10.180:22
>> $IPT -A FORWARD -p tcp --dport 22 -m state --state NEW -d 10.10.10.180 
>> -j ACCEPT
>>
>> This works if I ssh to the eth3, the dynamic dsl interface:
>>
>> ssh -p 2280 voip@<dsl ip address>
>>
>> I get an ssh session on the voip server.
>>
>> But:
>>
>> ssh -p 2280 voip@<static ip address>
>>
>> doesn't work. But I need to have others access the voip server using a 
>> static ip, but not give them access to the multihomed server.
> 
> 
> The ip rule won't work for reply packets sent by the server, because . 
> source address mangling occurs after the routing decision so the source 
> address is 10.10.10.180, not (yet) eth0's address. If Verizon drops 
> packets sent with a source address other than the one assigned to eth3, 
> then the client won't receive any reply and the connection will fail.
> 
> In order to route the reply packets using table 128, you need to 
> identify them. I guess that 10.10.10.180:22 as the source address:port 
> is not discriminant enough, as it matches connections forwarded from 
> eth3 too.
> 
> You can use the CONNMARK target to mark the incoming connection on eth0 
> and copy the connection mark to the reply packets on eth1. Then you can 
> use the packet mark in an ip rule.
> 
> iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
>   -j CONNMARK --set-mark 0x1
> iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark
> ip rule add fwmark 0x1 table 128 prio 127
> 
> As you used DNAT, you may use the --ctorigdst option of the 'conntrack' 
> match and mark reply packets based on the original destination address 
> of the connection.
> 
> iptables -t mangle -A PREROUTING -i eth1 \
>   -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
> ip rule add fwmark 0x1 table 128 prio 127


Thanks for the quick response. I, on the other hand, took some time off 
around xmas.

As always, it takes time for me to think through these ip/iptables 
problems. But I realized you'd pointed me in the direction of how to 
solve my general problem, not just ssh. I realized I could have all 
packets from the voip server go out the T1 interface quite simply:

#!/sh/bin
## eth0 is static to broadview
ETH0_IP_ADDR=www.xxx.yyy.zzz
ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
## this is the route through broadview gateway ip
ip route add default via <eth0 gateway ip> table 128

## this should make all packets from the * server go out over broadview
iptables -t mangle -A PREROUTING -i eth1 \
   -s 10.10.10.180 -j MARK --set-mark 0x1
## this is supposed to make all packets replying to eth0
## go out eth0
iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
    -j CONNMARK --set-mark 0x1
iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark
## this won't work on iptables 1.4.1
# iptables -t mangle -A PREROUTING -i eth1 \
#  -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
ip rule add fwmark 0x1 table 128 prio 127

so now all traffic, including the voip packets, from the voip server go 
out over the T1, but only the reply traffic from the rest of the lan 
goes out the T1.

As you can see ctorigdst didn't work with iptables in fedora 9:

iptables v1.4.1.1: Unknown arg `--ctorigdst`

which is puzzling, but...

Also, I see you set the new ip rule with priority 127. Am I right that 
higher priority numbers override lower priority number in case of a 
conflict? Or does it determine the order in which rules are applied, 
smaller numbers first?

Thanks for the help.

sean

Re: can't port forward on multihome

[sean darcy]

sean darcy wrote:
> Pascal Hambourg wrote:
>> Hello,
>>
>> sean darcy a écrit :
>>> I have a multihomed server: eth0 is a static T1, and eth3 is a 
>>> Verizon dsl line. I want eth3 as the default for general traffic, and 
>>> eth0 for VOIP traffic.
>>>
>>> eth1 is the internal interface. eth3 works fine as the masquerade out 
>>> for NAT'd lan.
>>>
>>> I've used ip to set up eth0 so I can ssh into it:
>>>
>>> ## eth0 is static
>>> ETH0_IP_ADDR=www.xxx.yyy.zzz
>>> ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
>>> ## this is the route through the gateway ip
>>> ip route add default via <eth0 gateway ip> table 128
>>>
>>> and that works. Which is important since that's the static address; 
>>> the Verizon dsl address is dynamic.
>>>
>>> The VOIP server ( asterisk ) is on the lan. I've tried to port 
>>> forward ssh to the voip server:
>>>
>>> $IPT -t nat -A PREROUTING  -p tcp --dport 2280 -j DNAT --to 
>>> 10.10.10.180:22
>>> $IPT -A FORWARD -p tcp --dport 22 -m state --state NEW -d 
>>> 10.10.10.180 -j ACCEPT
>>>
>>> This works if I ssh to the eth3, the dynamic dsl interface:
>>>
>>> ssh -p 2280 voip@<dsl ip address>
>>>
>>> I get an ssh session on the voip server.
>>>
>>> But:
>>>
>>> ssh -p 2280 voip@<static ip address>
>>>
>>> doesn't work. But I need to have others access the voip server using 
>>> a static ip, but not give them access to the multihomed server.
>>
>>
>> The ip rule won't work for reply packets sent by the server, because . 
>> source address mangling occurs after the routing decision so the 
>> source address is 10.10.10.180, not (yet) eth0's address. If Verizon 
>> drops packets sent with a source address other than the one assigned 
>> to eth3, then the client won't receive any reply and the connection 
>> will fail.
>>
>> In order to route the reply packets using table 128, you need to 
>> identify them. I guess that 10.10.10.180:22 as the source address:port 
>> is not discriminant enough, as it matches connections forwarded from 
>> eth3 too.
>>
>> You can use the CONNMARK target to mark the incoming connection on 
>> eth0 and copy the connection mark to the reply packets on eth1. Then 
>> you can use the packet mark in an ip rule.
>>
>> iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
>>   -j CONNMARK --set-mark 0x1
>> iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark
>> ip rule add fwmark 0x1 table 128 prio 127
>>
>> As you used DNAT, you may use the --ctorigdst option of the 
>> 'conntrack' match and mark reply packets based on the original 
>> destination address of the connection.
>>
>> iptables -t mangle -A PREROUTING -i eth1 \
>>   -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
>> ip rule add fwmark 0x1 table 128 prio 127
> 
> 
> Thanks for the quick response. I, on the other hand, took some time off 
> around xmas.
> 
> As always, it takes time for me to think through these ip/iptables 
> problems. But I realized you'd pointed me in the direction of how to 
> solve my general problem, not just ssh. I realized I could have all 
> packets from the voip server go out the T1 interface quite simply:
> 
> #!/sh/bin
> ## eth0 is static to broadview
> ETH0_IP_ADDR=www.xxx.yyy.zzz
> ip rule add from $ETH0_IP_ADDR/32 table 128 priority 128
> ## this is the route through broadview gateway ip
> ip route add default via <eth0 gateway ip> table 128
> 
> ## this should make all packets from the * server go out over broadview
> iptables -t mangle -A PREROUTING -i eth1 \
>   -s 10.10.10.180 -j MARK --set-mark 0x1
> ## this is supposed to make all packets replying to eth0
> ## go out eth0
> iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
>    -j CONNMARK --set-mark 0x1
> iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark
> ## this won't work on iptables 1.4.1
> # iptables -t mangle -A PREROUTING -i eth1 \
> #  -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
> ip rule add fwmark 0x1 table 128 prio 127
> 
> so now all traffic, including the voip packets, from the voip server go 
> out over the T1, but only the reply traffic from the rest of the lan 
> goes out the T1.
> 
> As you can see ctorigdst didn't work with iptables in fedora 9:
> 
> iptables v1.4.1.1: Unknown arg `--ctorigdst`
> 
> which is puzzling, but...
> 
> Also, I see you set the new ip rule with priority 127. Am I right that 
> higher priority numbers override lower priority number in case of a 
> conflict? Or does it determine the order in which rules are applied, 
> smaller numbers first?
> 
> Thanks for the help.
> 
> sean
> 
> 
Er, hit send too soon:

## this should make all packets from the * server go out over broadview
iptables -t mangle -A PREROUTING -i eth1 \
    -s 10.10.10.180 -j MARK --set-mark 0x1
ip rule add fwmark 0x1 table 128 prio 127


adding this work fine I can ssh and make voip connections to the voip 
server.

But if add this, I can no longer make a voip connection to the voip server.

## this is supposed to make all packets replying to eth0
## go out eth0
iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
     -j CONNMARK --set-mark 0x1
iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark

I also tried a new mark of 0x2 and an ip rule for 0x2 with a prio of 
126. That didn't work either.

So I'm only half-way there. What did I do wrong?

sean

Re: can't port forward on multihome

[Pascal Hambourg]

sean darcy a écrit :
> sean darcy wrote:
>> Pascal Hambourg wrote:
>>>
>>> As you used DNAT, you may use the --ctorigdst option of the 
>>> 'conntrack' match and mark reply packets based on the original 
>>> destination address of the connection.
>>>
>>> iptables -t mangle -A PREROUTING -i eth1 \
>>>   -m connmark --ctorigdst $ETH0_IP_ADDR -j MARK --set-mark 0x1
>>> ip rule add fwmark 0x1 table 128 prio 127
[...]
>> As you can see ctorigdst didn't work with iptables in fedora 9:
>>
>> iptables v1.4.1.1: Unknown arg `--ctorigdst`

My mistake, I meant "-m conntrack", not "-m connmark".

>> Also, I see you set the new ip rule with priority 127. Am I right that 
>> higher priority numbers override lower priority number in case of a 
>> conflict? Or does it determine the order in which rules are applied, 
>> smaller numbers first?

Rules with a lower priority number are examined first. I set a different 
priority because I thought there could be only one rule per priority, 
but I was wrong. However I don't know in which order rules with the same 
priority are examined, so setting different priorities may be safer if 
the rule ordering matters.

> ## this should make all packets from the * server go out over broadview
> iptables -t mangle -A PREROUTING -i eth1 \
>    -s 10.10.10.180 -j MARK --set-mark 0x1
> ip rule add fwmark 0x1 table 128 prio 127
> 
> adding this work fine I can ssh and make voip connections to the voip 
> server.
> 
> But if add this, I can no longer make a voip connection to the voip server.
> 
> ## this is supposed to make all packets replying to eth0
> ## go out eth0
> iptables -t mangle -A PREROUTING -i eth0 -m state --state NEW \
>     -j CONNMARK --set-mark 0x1
> iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --restore-mark

Which VoIP protocol are you using ? If it is SIP or H.323, are the 
corresponding conntrack and NAT helper modules loaded ? I believe they 
are required so that the connection tracking can set the connection mark 
to the related voice traffic.