[RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

Hi!

Apparently a lot of people still have issues when combining MASQUERADE
and policy routing.  

In those cases, the error path from net/ipv4/netfilter/ipt_MASQUERADE.c:

		if (rt->u.dst.dev != out) {
			if (net_ratelimit())
				printk("MASQUERADE:"
				       " Route sent us somewhere else.\n");
			ip_rt_put(rt);
			return NF_DROP;
		}

is taken.

A couple of questions:

1) Why do we have this check in the first place?  What would be wrong
   with re-routing if the user requests us by configuration?

2) Why don't we include 'oif' in the routing key?  If we wanted to make
   sure that oif doesn't change, then we should tell the routing lookup
   rather than complaining afterwards, shouldn't we ?

I think the current code makes it hard (if not impossible?) for the
system adminsitrator to get it right (for no apparent reason to me).

Comments, anyone?
-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[David S. Miller]

On Mon, 30 Aug 2004 22:19:57 +0200
Harald Welte <laforge@netfilter.org> wrote:

> In those cases, the error path from net/ipv4/netfilter/ipt_MASQUERADE.c:
 ...
> is taken.
> 
> A couple of questions:
> 
> 1) Why do we have this check in the first place?  What would be wrong
>    with re-routing if the user requests us by configuration?
> 
> 2) Why don't we include 'oif' in the routing key?  If we wanted to make
>    sure that oif doesn't change, then we should tell the routing lookup
>    rather than complaining afterwards, shouldn't we ?

The original idea was that if the input route relookup sends us
to a different device, something is very strange.

Input routing looks up using daddr/saddr/tos as the key.  The change
here is that we're now using the fwmark, so you're right that only
policy routing can cause this thing to trigger.

Where did this check some from?  From this change below, and the
changelog explains why we do things this way.

I know you don't use BK Harald, but things like this are why you should
at least use the web based interface to look at file change history.

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2003/08/11 22:47:11-07:00 rusty@rustcorp.com.au 
#   [NETFILTER]: Fix masquerade routing check.
#   
#   Alexey says:
#    Unrelated: giving out->ifindex is a bug, by the way. It can screw up
#    the things a lot. In this context, if you want to be sure that packet
#    will go out expected interface you do plain lookup and drop packet
#    if it gave you some strange route.
# 
# net/ipv4/netfilter/ipt_MASQUERADE.c
#   2003/08/11 22:46:31-07:00 rusty@rustcorp.com.au +11 -4
#   [NETFILTER]: Fix masquerade routing check.
# 
diff -Nru a/net/ipv4/netfilter/ipt_MASQUERADE.c b/net/ipv4/netfilter/ipt_MASQUERADE.c
--- a/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-30 13:51:03 -07:00
+++ b/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-30 13:51:03 -07:00
@@ -91,11 +91,18 @@
 #ifdef CONFIG_IP_ROUTE_FWMARK
 						.fwmark = (*pskb)->nfmark
 #endif
-					      } },
-				    .oif = out->ifindex };
+					      } } };
 		if (ip_route_output_key(&rt, &fl) != 0) {
-			/* Shouldn't happen */
-			printk("MASQUERADE: No route: Rusty's brain broke!\n");
+			/* Funky routing can do this. */
+			if (net_ratelimit())
+				printk("MASQUERADE:"
+				       " No route: Rusty's brain broke!\n");
+			return NF_DROP;
+		}
+		if (rt->u.dst.dev != out) {
+			if (net_ratelimit())
+				printk("MASQUERADE:"
+				       " Route sent us somewhere else.\n");
 			return NF_DROP;
 		}
 	}

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

Thanks for your quick reply, Dave.

On Mon, Aug 30, 2004 at 02:07:29PM -0700, David S. Miller wrote:

> The original idea was that if the input route relookup sends us
> to a different device, something is very strange.
> 
> Input routing looks up using daddr/saddr/tos as the key.  The change
> here is that we're now using the fwmark, so you're right that only
> policy routing can cause this thing to trigger.

and it apparently happens in a lot of 'typical' setups where you have a
masqueraded DSL line with dynamic ip address for bulk traffic, and
fwmark-based routing through one or more tunnel interfaces, optionally
with different masqerading/SNAT.

Also, the MASQUERADE lookup (obviously) has no more saddr in the lookup,
that's another difference from the 'original' lookup.

> Where did this check some from?  From this change below, and the
> changelog explains why we do things this way.
> 
> I know you don't use BK Harald, but things like this are why you should
> at least use the web based interface to look at file change history.

Thanks Dave, I did look up the bk web interface on this item before sending
this post ;) However

> # It can screw up the things a lot. 

does not really give me an understanding of why and where it might screw
up. I really want  to fully understand this issue before proposing any
change.

> # In this context, if you want to be sure that packet will go out
> # expected interface you do plain lookup and drop packet if it gave
> # you some strange route.

That is the presumption I am about to challenge.  Is the 'original'
interface really the one we want in this case? 

I've seen a number of users commenting out that check or even starting
to use the iptables ROUTE target (ugly) to get it working in their
setup.  Or they start to use SNAT with scripts in PPP if-up to update
the ruleset with the new dynamic IP :(

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[David S. Miller]

On Tue, 31 Aug 2004 03:38:42 +0200
Harald Welte <laforge@netfilter.org> wrote:

> Thanks for your quick reply, Dave.

Was this one quicker? :)

> > # It can screw up the things a lot. 
> 
> does not really give me an understanding of why and where it might screw
> up. I really want  to fully understand this issue before proposing any
> change.
> 
> > # In this context, if you want to be sure that packet will go out
> > # expected interface you do plain lookup and drop packet if it gave
> > # you some strange route.
> 
> That is the presumption I am about to challenge.  Is the 'original'
> interface really the one we want in this case? 

Good question.

What Alexey appears to be objecting to, exactly, is how Rusty added
a specific output device specifier to the flow key for route lookup.

I think the check can be removed.  If someone screams, we'll revisit.

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

Harald Welte <laforge@netfilter.org> wrote:
> 
> I've seen a number of users commenting out that check or even starting
> to use the iptables ROUTE target (ugly) to get it working in their
> setup.  Or they start to use SNAT with scripts in PPP if-up to update
> the ruleset with the new dynamic IP :(

Yes I had to convert all my MASQUERADE rules over to SNAT due to this
problem.  Unfortunately I had to convert them back again because SNAT
doesn't do an automatic flush which MASQUERADE does.  Without the flush
it's pretty useless when your interface address changes often.

So it would be good to know why the oif key is a bad idea.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[David S. Miller]

On Tue, 31 Aug 2004 12:20:53 +1000
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> So it would be good to know why the oif key is a bad idea.

I'm OK with removing the oif check, but I'm not OK with the
idea of specifying a specific output interface in the
route lookup flow key here.

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

David S. Miller <davem@davemloft.net> wrote:
> 
> Good question.
> 
> What Alexey appears to be objecting to, exactly, is how Rusty added
> a specific output device specifier to the flow key for route lookup.
> 
> I think the check can be removed.  If someone screams, we'll revisit.

I don't think removing the check fixes the problem.  If the route
does go to a different interface the source address may be completely
bogus for the original packet.

This is what happens:

* Forwarded packet is routed to iface1.
* Packet hits MASQUERADE.
* Routing lookup returns iface2 with different source address.

So if iface2's source address is not valid when the packet leaves on
iface1, then the packet won't go very far.

If you're wondering why the second lookup is returning a different
interface at all, it's because the routing lookup in MASQUERADE is
done as if the packet was generated by localhost.  This is obviously
going to differ from the normal routing lookup if the packet was
forwarded.

The most common scenario for this is multiple nexthops in your
default route.  The first lookup will return iface1, but the
second lookup may return a different nexthop.

As it stands that packet is dropped which is no good.  But removing
the check just sends the packet out of iface1 with iface2's address
which isn't much better depending on your ISP's filtering policy.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Mon, Aug 30, 2004 at 07:24:58PM -0700, David S. Miller wrote:
> On Tue, 31 Aug 2004 12:20:53 +1000
> Herbert Xu <herbert@gondor.apana.org.au> wrote:
> 
> > So it would be good to know why the oif key is a bad idea.
> 
> I'm OK with removing the oif check, but I'm not OK with the
> idea of specifying a specific output interface in the
> route lookup flow key here.

Can you please give a reason for that?
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[David S. Miller]

On Tue, 31 Aug 2004 12:32:40 +1000
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> This is what happens:
> 
> * Forwarded packet is routed to iface1.
> * Packet hits MASQUERADE.
> * Routing lookup returns iface2 with different source address.
> 
> So if iface2's source address is not valid when the packet leaves on
> iface1, then the packet won't go very far.
> 
> If you're wondering why the second lookup is returning a different
> interface at all, it's because the routing lookup in MASQUERADE is
> done as if the packet was generated by localhost.  This is obviously
> going to differ from the normal routing lookup if the packet was
> forwarded.

I understand this description.

Would it be enough to set 'out' to rt->u.dst.dev after the call to
ip_route_output_key() in ipt_MASQUERADE.c?

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Mon, Aug 30, 2004 at 10:39:20PM -0700, David S. Miller wrote:
>
> > If you're wondering why the second lookup is returning a different
> > interface at all, it's because the routing lookup in MASQUERADE is
> > done as if the packet was generated by localhost.  This is obviously
> > going to differ from the normal routing lookup if the packet was
> > forwarded.
> 
> I understand this description.
> 
> Would it be enough to set 'out' to rt->u.dst.dev after the call to
> ip_route_output_key() in ipt_MASQUERADE.c?

Unfortunately no.  You see the result of ip_route_output_key() is
not used at all so you can set rt->u.dst.dev all you like and it
isn't going to make any difference :)

The only reason we're making that call in MASQUERADE is so that we
can get the source address from the resulting dst.

Now if the dst is pointing to the wrong device, then that source
address may be invalid.

In fact I wonder if Alexey was aware of the fact that this dst is
not going to be used for routing the packet.  It's thrown away
immediately after we get the source address.

In this case I really don't see what's wrong with setting oif in
the key.  In fact that's exactly what we want to do: Get the
preferred source address when the localhost is going to send
a packet to *that* device.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Julian Anastasov]

	Hello,

On Mon, 30 Aug 2004, David S. Miller wrote:

> > If you're wondering why the second lookup is returning a different
> > interface at all, it's because the routing lookup in MASQUERADE is
> > done as if the packet was generated by localhost.  This is obviously
> > going to differ from the normal routing lookup if the packet was
> > forwarded.
>
> I understand this description.
>
> Would it be enough to set 'out' to rt->u.dst.dev after the call to
> ip_route_output_key() in ipt_MASQUERADE.c?

	I think, the picture is:

Packet 1:

- input route (before NAT) => dev1

- MASQUERADE => autoselect public IP, try to use dev1 (same GW)
as already selected from input route

Packet 2..n:

- input route (before NAT) => cached dev1 (until routing/cache changes)

	So, if the input route for all packets selects dev1 before
NAT but MASQUERADE selects different device (nexthop) bad things
happen. It costs routing cache entries to provide oif key but almost
in any case the right gateway is selected (except when two nexthops
use same device).

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Tue, Aug 31, 2004 at 04:43:37PM +1000, herbert wrote:
> 
> In this case I really don't see what's wrong with setting oif in
> the key.  In fact that's exactly what we want to do: Get the
> preferred source address when the localhost is going to send
> a packet to *that* device.

Here's a better idea.  Why don't we forget about the route lookup
and call inet_select_addr directly?

The only time when someone wants to use MASQUERADE over SNAT is
when the interface carries a dynamic address.  In such cases,
inet_select_addr should always return the preferred address, no?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

On Tue, Aug 31, 2004 at 05:00:43PM +1000, herbert wrote:
> 
> Here's a better idea.  Why don't we forget about the route lookup
> and call inet_select_addr directly?

Even better, why don't we use the skb->dst directly? It already
contains the preferred source for the route that led us to the
device.

Here is an untested patch which does exactly that? Please do your
best to break it :)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[-- Attachment #2: p --]
[-- Type: text/plain, Size: 1162 bytes --]

===== net/ipv4/netfilter/ipt_MASQUERADE.c 1.16 vs edited =====
--- 1.16/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-02 12:27:11 +10:00
+++ edited/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-31 18:21:41 +10:00
@@ -97,34 +97,13 @@
 
 	mr = targinfo;
 
-	{
-		struct flowi fl = { .nl_u = { .ip4_u =
-					      { .daddr = (*pskb)->nh.iph->daddr,
-						.tos = (RT_TOS((*pskb)->nh.iph->tos) |
-							RTO_CONN),
-#ifdef CONFIG_IP_ROUTE_FWMARK
-						.fwmark = (*pskb)->nfmark
-#endif
-					      } } };
-		if (ip_route_output_key(&rt, &fl) != 0) {
-			/* Funky routing can do this. */
-			if (net_ratelimit())
-				printk("MASQUERADE:"
-				       " No route: Rusty's brain broke!\n");
-			return NF_DROP;
-		}
-		if (rt->u.dst.dev != out) {
-			if (net_ratelimit())
-				printk("MASQUERADE:"
-				       " Route sent us somewhere else.\n");
-			ip_rt_put(rt);
-			return NF_DROP;
-		}
-	}
+	rt = (struct rtable *)(*pskb)->dst;
+	if (rt->fl.iif)
+		newsrc = rt->rt_spec_dst;
+	else
+		newsrc = rt->rt_src;
 
-	newsrc = rt->rt_src;
 	DEBUGP("newsrc = %u.%u.%u.%u\n", NIPQUAD(newsrc));
-	ip_rt_put(rt);
 
 	WRITE_LOCK(&masq_lock);
 	ct->nat.masq_index = out->ifindex;

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Julian Anastasov]

	Hello,

On Tue, 31 Aug 2004, Herbert Xu wrote:

> On Tue, Aug 31, 2004 at 05:00:43PM +1000, herbert wrote:
> >
> > Here's a better idea.  Why don't we forget about the route lookup
> > and call inet_select_addr directly?
>
> Even better, why don't we use the skb->dst directly? It already
> contains the preferred source for the route that led us to the
> device.
>
> Here is an untested patch which does exactly that? Please do your
> best to break it :)

	No need for tests :) rt_spec_dst is your preferred src
to the sender (your local internal IP) and rt_src is the internal
IP of the sender (what we snat).

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Tue, Aug 31, 2004 at 12:41:35PM +0300, Julian Anastasov wrote:
>
> > Here is an untested patch which does exactly that? Please do your
> > best to break it :)
> 
> 	No need for tests :) rt_spec_dst is your preferred src
> to the sender (your local internal IP) and rt_src is the internal
> IP of the sender (what we snat).

I knew it can't be that easy :)

So let's go back to the previous idea of using inet_select_addr?
Can you find any problems with this? In fact, it seems that the
2.2 compatibility code does exactly this.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[-- Attachment #2: p --]
[-- Type: text/plain, Size: 1183 bytes --]

===== net/ipv4/netfilter/ipt_MASQUERADE.c 1.16 vs edited =====
--- 1.16/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-02 12:27:11 +10:00
+++ edited/net/ipv4/netfilter/ipt_MASQUERADE.c	2004-08-31 20:08:47 +10:00
@@ -97,34 +97,12 @@
 
 	mr = targinfo;
 
-	{
-		struct flowi fl = { .nl_u = { .ip4_u =
-					      { .daddr = (*pskb)->nh.iph->daddr,
-						.tos = (RT_TOS((*pskb)->nh.iph->tos) |
-							RTO_CONN),
-#ifdef CONFIG_IP_ROUTE_FWMARK
-						.fwmark = (*pskb)->nfmark
-#endif
-					      } } };
-		if (ip_route_output_key(&rt, &fl) != 0) {
-			/* Funky routing can do this. */
-			if (net_ratelimit())
-				printk("MASQUERADE:"
-				       " No route: Rusty's brain broke!\n");
-			return NF_DROP;
-		}
-		if (rt->u.dst.dev != out) {
-			if (net_ratelimit())
-				printk("MASQUERADE:"
-				       " Route sent us somewhere else.\n");
-			ip_rt_put(rt);
-			return NF_DROP;
-		}
-	}
+	rt = (struct rtable *)(*pskb)->dst;
+	newsrc = inet_select_addr(out, rt->rt_gateway, RT_SCOPE_LINK);
+	if (!newsrc)
+		return NF_DROP;
 
-	newsrc = rt->rt_src;
 	DEBUGP("newsrc = %u.%u.%u.%u\n", NIPQUAD(newsrc));
-	ip_rt_put(rt);
 
 	WRITE_LOCK(&masq_lock);
 	ct->nat.masq_index = out->ifindex;

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Henrik Nordstrom]

On Tue, 31 Aug 2004, Harald Welte wrote:

> and it apparently happens in a lot of 'typical' setups where you have a
> masqueraded DSL line with dynamic ip address for bulk traffic, and
> fwmark-based routing through one or more tunnel interfaces, optionally
> with different masqerading/SNAT.

Should only happen if the routing is screwed up, in principle..

> Also, the MASQUERADE lookup (obviously) has no more saddr in the lookup,
> that's another difference from the 'original' lookup.

This is a possible reason why it screws up for such many people? By not 
including the source IP you make the route lookup for determining the 
MASQUERADE information very different from the route lookup when 
forwardning the packet.

I think it would for most make more sense that the source IP assignment is 
based on routing using the original source address as key.

What I don't get is why MASQUERADE needs to do a route lookup at all. 
Isn't the traffic already routed at this point, and shouldn't it simply 
use the source IP of the current selected route or the interface primary 
IP of no source IP set in the route?

> That is the presumption I am about to challenge.  Is the 'original'
> interface really the one we want in this case?

If there is policy routing saying that packets with a given source 
should go out another interface my opinion is that they should.

I have been using SNAT in quite many complex routing setups and so far has 
not run into any situation where routing gets wrong in an unmanageable and 
unintuitive manner due to SNAT. MASQUERADE should not be more complex.

Regards
Henrik

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Julian Anastasov]

	Hello,

On Tue, 31 Aug 2004, Herbert Xu wrote:

> I knew it can't be that easy :)
>
> So let's go back to the previous idea of using inet_select_addr?
> Can you find any problems with this? In fact, it seems that the
> 2.2 compatibility code does exactly this.

	Yes, 2.2 worked somehow, until some users wanted more
control from the routing. Now ip_fw_compat_masq.c has such call,
it can be fast in some setups with small number of IPs but may
be now there some setups using nfmark for selecting maddr from
ISP-specific routing tables. I'm not sure we can avoid the routing
and the cache pollution, we have to live with the old behavoir, not
the current one.

	I have some related work, you can check it for more ideas:

http://www.ssi.bg/~ja/#routes
see dgd-usage.txt

	In short, I use the mpath route as a nexthop selector (on
first packet), the next packets do not hit the mpath route, they feed the 
routing with the maddr already bound to the NAT connection. This survives
cache expiration. May be using connmark is a better idea:
assign nfmark to connection and route it always via same GW. But
may be the oif key remains, I'm not sure.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Tue, Aug 31, 2004 at 01:54:21PM +0300, Julian Anastasov wrote:
> 
> 	Yes, 2.2 worked somehow, until some users wanted more
> control from the routing. Now ip_fw_compat_masq.c has such call,
> it can be fast in some setups with small number of IPs but may
> be now there some setups using nfmark for selecting maddr from
> ISP-specific routing tables. I'm not sure we can avoid the routing
> and the cache pollution, we have to live with the old behavoir, not
> the current one.

You're right.  However, we can still do this without performing
another routing lookup.  The information is already there in the
fib rule.  We just didn't bother writing it down in the route for
forwarded packets.

So if we add a new field for the preferred source address to
struct rtable then we can avoid the lookup.

BTW, I'd still like to know the problem with the original oif key.
It's basically saying that if you can find the correct route using
the other keys (daddr/tos/mark) then all is well, if you can't
(dev != out) then we'll use the best address on the outgoing
interface.

> 	I have some related work, you can check it for more ideas:
> 
> http://www.ssi.bg/~ja/#routes
> see dgd-usage.txt
> 
> 	In short, I use the mpath route as a nexthop selector (on
> first packet), the next packets do not hit the mpath route, they feed the 
> routing with the maddr already bound to the NAT connection. This survives
> cache expiration. May be using connmark is a better idea:
> assign nfmark to connection and route it always via same GW. But
> may be the oif key remains, I'm not sure.

Fortunately I'm one of those with lots of interfaces but only one
IP on each.  I can sympathise with your situation of many IPs but
only one interface :)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

Henrik Nordstrom <hno@marasystems.com> wrote:
> 
> Should only happen if the routing is screwed up, in principle..

If only that were the case :)
 
> I think it would for most make more sense that the source IP assignment is 
> based on routing using the original source address as key.

Doesn't work because we're trying to get the correct source address from
the routing lookup.

> What I don't get is why MASQUERADE needs to do a route lookup at all. 
> Isn't the traffic already routed at this point, and shouldn't it simply 
> use the source IP of the current selected route or the interface primary 
> IP of no source IP set in the route?

The problem is that we don't record the preferred source in struct rtable.
It is available in the original FIB rule.  So by adding a new field in
struct rtable we should be able to avoid this lookup.

>> That is the presumption I am about to challenge.  Is the 'original'
>> interface really the one we want in this case?
> 
> If there is policy routing saying that packets with a given source 
> should go out another interface my opinion is that they should.

Absolutely.

> I have been using SNAT in quite many complex routing setups and so far has 
> not run into any situation where routing gets wrong in an unmanageable and 
> unintuitive manner due to SNAT. MASQUERADE should not be more complex.

Well I took that advice and ended up with tons of broken systems because
SNAT is totally broken when your IP address is dynamic.  MASQUERADE
flushes the conntrack rules automatically.  SNAT does not.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Tue, Aug 31, 2004 at 09:48:07AM +0300, Julian Anastasov wrote:

> 	So, if the input route for all packets selects dev1 before
> NAT but MASQUERADE selects different device (nexthop) bad things
> happen. It costs routing cache entries to provide oif key but almost
> in any case the right gateway is selected (except when two nexthops
> use same device).

I am willing to compromise at that cost.  I cannot imagine a combination
of dynamic IP with multiple nexthop on the same device.  Getting those
policy routing / DSL / dynip / MASQUERADE cases right is definitely more
important.

Any static IP case should be using SNAT, that's always been documented.

> Regards

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

On Tue, Aug 31, 2004 at 12:24:34PM +0200, Henrik Nordstrom wrote:
 
> >and it apparently happens in a lot of 'typical' setups where you have a
> 
> Should only happen if the routing is screwed up, in principle..

yes, but it happens to happen more often, thus I see it as a bug.

> >Also, the MASQUERADE lookup (obviously) has no more saddr in the lookup,
> >that's another difference from the 'original' lookup.
> 
> This is a possible reason why it screws up for such many people? By not 
> including the source IP you make the route lookup for determining the 
> MASQUERADE information very different from the route lookup when 
> forwardning the packet.

Yes, since it now looks to the routing code as if you wanted to find out
a source ip for locally-originated packets.

> I think it would for most make more sense that the source IP assignment is 
> based on routing using the original source address as key.

Question is: can we do this?  Can we ask the routing code to choose a
source address while we already specify one?  I don't think so.

> >That is the presumption I am about to challenge.  Is the 'original'
> >interface really the one we want in this case?
> 
> If there is policy routing saying that packets with a given source 
> should go out another interface my opinion is that they should.

Ok, I think I agree with you. It sounds like the right thing to do,
rather than trying to fix a broken configuration within MASQUERADE.

> Regards
> Henrik

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Julian Anastasov]

	Hello,

On Tue, 31 Aug 2004, Herbert Xu wrote:

> On Tue, Aug 31, 2004 at 01:54:21PM +0300, Julian Anastasov wrote:
> >
> > 	Yes, 2.2 worked somehow, until some users wanted more
> > control from the routing. Now ip_fw_compat_masq.c has such call,
> > it can be fast in some setups with small number of IPs but may
> > be now there some setups using nfmark for selecting maddr from
> > ISP-specific routing tables. I'm not sure we can avoid the routing
> > and the cache pollution, we have to live with the old behavoir, not
> > the current one.
>
> You're right.  However, we can still do this without performing
> another routing lookup.  The information is already there in the
> fib rule.  We just didn't bother writing it down in the route for
> forwarded packets.

	I do not see where the public IP is, what you mean? As the
mpath route does not have preferred src IP (usually when many ISPs
are used) the kernel uses inet_select_addr to select one, in similar
way as you are trying to do. But the difference is that it is now
cached and by using nfmark we have more options not to reach this
mpath route on next lookups.

	One example: the GWs used in the nexthops can be internal
addresses, in such cases inet_select_addr gets the first local (scope
global) public IP address because the target IP is not always part
from the GW's subnet. You can not rely on any information present
in the mpath route and to assume anything about maddr without specific
lookup, additional one.

> So if we add a new field for the preferred source address to
> struct rtable then we can avoid the lookup.

	What is this? Sort of 'nexthop via GW1 dev eth0 snat MADDR1' ?
This is the same as to use SNAT target. Similar thing worked very
well in 2.2 kernels (nat XXX) but only for unipath routes.

> BTW, I'd still like to know the problem with the original oif key.

	The old way to provide oif as key adds one additional
cache entry per every normal input route. Another issue is that
providing oif key can hit wrong route in some setups - not the
first match which we usually hit with oif=0. But for the usual
cases it works.

> It's basically saying that if you can find the correct route using
> the other keys (daddr/tos/mark) then all is well, if you can't
> (dev != out) then we'll use the best address on the outgoing
> interface.

	ip_route_output + inet_select_addr in 50% of the cases
for mpath route with 2 NHs?

	inet_select_addr is ok to use only when nfmark is not used,
it is even used now for multipath routes in the usual case when
prefsrc is not defined. But with the routing you have more control
on what to select as maddr. So, I'm for the old way which can
work for more setups and against inet_select_addr which can break 
existing ones.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Tue, Aug 31, 2004 at 03:33:22PM +0300, Julian Anastasov wrote:
> 
> 	I do not see where the public IP is, what you mean? As the
> mpath route does not have preferred src IP (usually when many ISPs
> are used) the kernel uses inet_select_addr to select one, in similar
> way as you are trying to do. But the difference is that it is now
> cached and by using nfmark we have more options not to reach this
> mpath route on next lookups.

I was mistaken.  In the mpath case there is no source address per
nexthop.
 
> 	The old way to provide oif as key adds one additional
> cache entry per every normal input route. Another issue is that

You still have one extra cache entry per input route now as the
keys of this lookup is not the same as the one for the packet.

> providing oif key can hit wrong route in some setups - not the
> first match which we usually hit with oif=0. But for the usual
> cases it works.

True.  But this has only been the case since a year ago.  And I
agree with others that if you really need this, then SNAT is
better since your addresses are likely to be static and you
can use the mark directly in netfilter.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Herbert Xu]

On Wed, Sep 01, 2004 at 07:28:02AM +1000, herbert wrote:
> On Tue, Aug 31, 2004 at 03:33:22PM +0300, Julian Anastasov wrote:
> > 
> > 	I do not see where the public IP is, what you mean? As the
> > mpath route does not have preferred src IP (usually when many ISPs
> > are used) the kernel uses inet_select_addr to select one, in similar
> > way as you are trying to do. But the difference is that it is now
> > cached and by using nfmark we have more options not to reach this
> > mpath route on next lookups.
> 
> I was mistaken.  In the mpath case there is no source address per
> nexthop.

Actually, that should still work.

For example, if you're like me and the nexthops all go to different
devices then it's obviously OK as inet_select_addr will pick the
right one for the device.  If they're going through the same device
but to different gateways then it'll still pick the right one for
the given gateway.

It only breaks when all your nexthops go to the same gateway (can't
happen) or if somehow the gateway addresses don't match up with your
desired source address.

In that case I really think that you should use SNAT :)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [RFC] MASQUERADE / policy routing ("Route send us somewhere else")

[Julian Anastasov]

	Hello,

On Wed, 1 Sep 2004, Herbert Xu wrote:

> > I was mistaken.  In the mpath case there is no source address per
> > nexthop.
>
> Actually, that should still work.
>
> For example, if you're like me and the nexthops all go to different
> devices then it's obviously OK as inet_select_addr will pick the
> right one for the device.  If they're going through the same device
> but to different gateways then it'll still pick the right one for
> the given gateway.

	Yes, if the targets are from some of the GW's subnets. It
is a masquerade drawback not to match by GW because the routing
does not support it but the world is not perfect.

Regards

--
Julian Anastasov <ja@ssi.bg>