Problem using fwmarks as routing key: "MASQUERADE: Route sent us somewhere else."

Problem using fwmarks as routing key: "MASQUERADE: Route sent us somewhere else."

[Thhoep]

hi,

my aim: to divide 100 hosts upon 6 masqueraded adsl connections to the
internet using a linux router runnig a debian woody.

the problem: really strange behaviour of the routing/masquerading combo,
that changes with every tried kernel version. (described below)

presumption: some version mismatch or a bug in the kernel routing code,
which needs a bugfix that till now is unknown to me

my config:

masquerading is activated in "/etc/ppp/ip-up.d" using
-----------
iptables -t nat -A POSTROUTING -o $PPP_IFACE -j MASQUERADE
-----------

routing policy is defined upon reboot:
-----------
echo 1 > /proc/sys/net/ipv4/ip_forward

ip route flush table main
ip route add 192.168.1.0/24 dev eth4 table main

# routing policies for adsl
for (( i=1 ; $i<7; i++ )) ; do
    ip route flush table dsl$i
    ip route add 192.168.1.0/24 dev eth4 table dsl$i
    ip rule add fwmark $i table dsl$i
done

# dividing users
iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.10 -j MARK --set-mark
1
iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.11 -j MARK --set-mark
1
#....
iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.17 -j MARK --set-mark
2
#....  and so on
-----------

external default routes are added to the dsl? tables in "/etc/ppp/ip-up.d"
using
-----------
ip route add $PPP_REMOTE dev $PPP_IFACE src $PPP_LOCAL table $PPP_IPPARAM
ip route add default via $PPP_REMOTE dev $PPP_IFACE table $PPP_IPPARAM
ip route flush cache
-----------

versions:
kernels tried: 2.4.20 - 2.6.1
iptables 1.2.9
iproute iproute2-ss010824
pppd version 2.4.1
patches: none

PROBLEM DESCRIPTION:

if i do a ping from an internal host to an external host i can see with
tcpdump, that the ping request found its way out of ppp?, a reply is coming
in, but isn't sent out of the internal interface eth4. if i use pure
counting netfilter rules for debugging i see, that the replies get lost
between PREROUTING and FORWARD, so i assume they get lost while routing. a
"route -C -n" shows a route to the external host, but no route back from the
external to the internal. in "/var/log/kern.log" are the only
networking-related messages:
-----------
kernel: request_module: failed /sbin/modprobe -- net-pf-10. error = 256
kernel: MASQUERADE: Route sent us somewhere else.
-----------
net-pf-10 seems to be an alias for some ipv6 module. i don't use ipv6 and
therefor don't have this module. so i ignored that error. that maquerading
message could be related to my problem. formerly with an older but very
similar configuration (i think just the iptables was linked to an older
kernel, but version was the same) i got another message:
-----------
kernel: MASQUERADE: No route: Rusty's brain broke!
-----------
i can't interpret any of both.

in addition (this seems to be important): in an older configuration i didnt
use fwmarks to select the routing tables but the inbuilt "from <ip>"
directive of the "ip"-tool like this:
-----------
ip rule add from 192.168.1.10 table dsl1
-----------
and in this configuration the router WAS WORKING absolutely nice and exactly
as expected. my problem only appears when i try to use fwmarks as routing
key. (the corresponding kernal feature is enabled).

please help me! i tried very hard and very long to solve this problem (over
weeks) and if i cant solve it i have to try it with openbsd or something
like that, out of pure desperation..

with best regards,
thomas hoeppler

Re: Problem using fwmarks as routing key: "MASQUERADE: Route sent us somewhere else."

[Ramin Dousti]

I think you would get a better response at lartc.org mailing list.
Hope you solve your problem soon.

Ramin

On Wed, Jan 21, 2004 at 06:25:22PM +0100, Thhoep wrote:

> hi,
> 
> my aim: to divide 100 hosts upon 6 masqueraded adsl connections to the
> internet using a linux router runnig a debian woody.
> 
> the problem: really strange behaviour of the routing/masquerading combo,
> that changes with every tried kernel version. (described below)
> 
> presumption: some version mismatch or a bug in the kernel routing code,
> which needs a bugfix that till now is unknown to me
> 
> my config:
> 
> masquerading is activated in "/etc/ppp/ip-up.d" using
> -----------
> iptables -t nat -A POSTROUTING -o $PPP_IFACE -j MASQUERADE
> -----------
> 
> routing policy is defined upon reboot:
> -----------
> echo 1 > /proc/sys/net/ipv4/ip_forward
> 
> ip route flush table main
> ip route add 192.168.1.0/24 dev eth4 table main
> 
> # routing policies for adsl
> for (( i=1 ; $i<7; i++ )) ; do
>     ip route flush table dsl$i
>     ip route add 192.168.1.0/24 dev eth4 table dsl$i
>     ip rule add fwmark $i table dsl$i
> done
> 
> # dividing users
> iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.10 -j MARK --set-mark
> 1
> iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.11 -j MARK --set-mark
> 1
> #....
> iptables -t mangle -A PREROUTING -i eth4 -s 192.168.1.17 -j MARK --set-mark
> 2
> #....  and so on
> -----------
> 
> external default routes are added to the dsl? tables in "/etc/ppp/ip-up.d"
> using
> -----------
> ip route add $PPP_REMOTE dev $PPP_IFACE src $PPP_LOCAL table $PPP_IPPARAM
> ip route add default via $PPP_REMOTE dev $PPP_IFACE table $PPP_IPPARAM
> ip route flush cache
> -----------
> 
> versions:
> kernels tried: 2.4.20 - 2.6.1
> iptables 1.2.9
> iproute iproute2-ss010824
> pppd version 2.4.1
> patches: none
> 
> PROBLEM DESCRIPTION:
> 
> if i do a ping from an internal host to an external host i can see with
> tcpdump, that the ping request found its way out of ppp?, a reply is coming
> in, but isn't sent out of the internal interface eth4. if i use pure
> counting netfilter rules for debugging i see, that the replies get lost
> between PREROUTING and FORWARD, so i assume they get lost while routing. a
> "route -C -n" shows a route to the external host, but no route back from the
> external to the internal. in "/var/log/kern.log" are the only
> networking-related messages:
> -----------
> kernel: request_module: failed /sbin/modprobe -- net-pf-10. error = 256
> kernel: MASQUERADE: Route sent us somewhere else.
> -----------
> net-pf-10 seems to be an alias for some ipv6 module. i don't use ipv6 and
> therefor don't have this module. so i ignored that error. that maquerading
> message could be related to my problem. formerly with an older but very
> similar configuration (i think just the iptables was linked to an older
> kernel, but version was the same) i got another message:
> -----------
> kernel: MASQUERADE: No route: Rusty's brain broke!
> -----------
> i can't interpret any of both.
> 
> in addition (this seems to be important): in an older configuration i didnt
> use fwmarks to select the routing tables but the inbuilt "from <ip>"
> directive of the "ip"-tool like this:
> -----------
> ip rule add from 192.168.1.10 table dsl1
> -----------
> and in this configuration the router WAS WORKING absolutely nice and exactly
> as expected. my problem only appears when i try to use fwmarks as routing
> key. (the corresponding kernal feature is enabled).
> 
> please help me! i tried very hard and very long to solve this problem (over
> weeks) and if i cant solve it i have to try it with openbsd or something
> like that, out of pure desperation..
> 
> with best regards,
> thomas hoeppler
> 

Re: Problem using fwmarks as routing key: "MASQUERADE: Route sent us somewhere else."

[Rodrigo Severo]

Thhoep,

>On Wed, Jan 21, 2004 at 06:25:22PM +0100, Thhoep wrote:
>
>
>>-----------
>>kernel: request_module: failed /sbin/modprobe -- net-pf-10. error = 256
>>kernel: MASQUERADE: Route sent us somewhere else.
>>
>>
I think you can easily solve your problem substituting MASQUERADE with
SNAT. There is one upside and one downside to this solution.

The downside is the extra work you will have setting a create/delete
updated SNAT each time an ADSL goes up/down.

The upside is that it actually works :)

I say so because I have a working 2 ADSL connections + 1 fixed
connection setup working fine with the MARK + ip rule + SNAT mix.


I hope it helps you,

Rodrigo Severo

--
----------------------------------------------------
Rodrigo Severo
Fábrica de Idéias
Fone: +55(61)321 1357
Fax: +55(61)223 1712
SBS - Quadra 2 - Ed. Empire Center - Sala 1301
Brasília/DF - Brasil
CEP: 70.070-904
----------------------------------------------------

Re: Problem using fwmarks as routing key: "MASQUERADE: Route sent us somewhere else."

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

On Wed, Jan 21, 2004 at 06:25:22PM +0100, Thhoep wrote:
> hi,
> 
> my aim: to divide 100 hosts upon 6 masqueraded adsl connections to the
> internet using a linux router runnig a debian woody.
> 
> the problem: really strange behaviour of the routing/masquerading combo,
> that changes with every tried kernel version. (described below)
> 
> presumption: some version mismatch or a bug in the kernel routing code,
> which needs a bugfix that till now is unknown to me

Please read the recent archives of the netdev list with regard to
"Rusty's brain broke".  Also see 
https://bugzilla.netfilter.org/cgi-bin/bugzilla/show_bug.cgi?id=144

Alexey heavily argued against us reverting that change, however :(

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]