policy routing on locally generated packets

policy routing on locally generated packets

[lartc]

Hello Gentle Developers,

May I first start by extending my sincerest gratitude to all that have
particpated in the netfilter project -- needless to say that it has been
of infinite value to me and many, many users. Thanks, thanks, thanks.


So, there has been some traffic over a period of time on the lartc list
concerning application based routing, which translates more specifically
to the issue of how to do policy based routing of locally generated
packets given that the "routing decision" has been made prior to a
packet traversing the OUTPUT chain.
	
Take for example the following script:

#!/bin/bash
iptables --append OUTPUT --table mangle --match owner \
     --gid-owner 500 --jump MARK --set-mark 0x2

iptables --append OUTPUT --table mangle --match owner \
     --gid-owner 501 --jump MARK --set-mark 0x3

ip rule add fwmark 0x2 table 2
ip route add default dev eth0 src 192.168.0.1 via 192.168.0.254 table 2
ip rule add fwmark 0x3 table 3
ip route add default dev eth0 src 192.168.0.2 via 192.168.0.253 table 3
ip route flush cache
#end_script


This is a sample of source ip routing policy, but one could imagine
routing based on tos, etc. Regrettably, this example will not work as
the default routing table is consulted prior to the marking.

While I'm not a programmer but have dreamt up a new target for the
POSTROUTING chain NAT table:

iptables --append POSTROUTING --table nat --jump rt_lookup


The statement would force a new routing decision to be made using the
properties of the packet as it traverses the nat table, respecting any
marking, tos, etc.

Could any of you be kind enough to offer an analysis of such a new
target, i.e. would it work, would it be terribly complex, ...


Sincerely Yours,

Charles Shick

Re: policy routing on locally generated packets

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

lartc@manchotnetworks.net wrote:

>The statement would force a new routing decision to be made using the
>properties of the packet as it traverses the nat table, respecting any
>marking, tos, etc.
>
>Could any of you be kind enough to offer an analysis of such a new
>target, i.e. would it work, would it be terribly complex, ...
>

A new lookup is performed for outgoing packet if any key affecting
routing decision is changed. However source address is not influenced
by this lookup, for locally originating packets it is set to rt_src in
ip_queue_xmit and ip_build_and_send_pkt but it doesn't happen
automatically with a new lookup. I'm not sure if it would be possible
to set a new source address, sockets which are bound to interfaces
might not accept packets anymore. You could try this patch, it changes
the packets source after the routing key changed, please make sure
both unbound and bound sockets work correctly.

Bye
Patrick

>
>
>Sincerely Yours,
>
>Charles Shick
>
>
>
>
>
>
>  
>

[-- Attachment #2: x.diff --]
[-- Type: text/plain, Size: 399 bytes --]

===== net/core/netfilter.c 1.23 vs edited =====
--- 1.23/net/core/netfilter.c	Thu Jul 31 01:57:26 2003
+++ edited/net/core/netfilter.c	Mon Aug 25 14:25:19 2003
@@ -645,6 +645,7 @@
 		/* Drop old route. */
 		dst_release((*pskb)->dst);
 		(*pskb)->dst = &rt->u.dst;
+		iph->saddr = rt->rt_src;
 	} else {
 		/* non-local src, find valid iif to satisfy
 		 * rp-filter when calling ip_route_input. */

Re: policy routing on locally generated packets

[Patrick McHardy]

I don't know what i was thinking, this patch is going to confuse
all kinds of stuff (especially conntrack & nat). Sorry.
Patrick

Patrick McHardy wrote:

> lartc@manchotnetworks.net wrote:
>
>> The statement would force a new routing decision to be made using the
>> properties of the packet as it traverses the nat table, respecting any
>> marking, tos, etc.
>>
>> Could any of you be kind enough to offer an analysis of such a new
>> target, i.e. would it work, would it be terribly complex, ...
>>
>
> A new lookup is performed for outgoing packet if any key affecting
> routing decision is changed. However source address is not influenced
> by this lookup, for locally originating packets it is set to rt_src in
> ip_queue_xmit and ip_build_and_send_pkt but it doesn't happen
> automatically with a new lookup. I'm not sure if it would be possible
> to set a new source address, sockets which are bound to interfaces
> might not accept packets anymore. You could try this patch, it changes
> the packets source after the routing key changed, please make sure
> both unbound and bound sockets work correctly.
>
> Bye
> Patrick
>
>>
>>
>> Sincerely Yours,
>>
>> Charles Shick
>>
>>
>>
>>
>>
>>
>>  
>>
>------------------------------------------------------------------------
>
>===== net/core/netfilter.c 1.23 vs edited =====
>--- 1.23/net/core/netfilter.c	Thu Jul 31 01:57:26 2003
>+++ edited/net/core/netfilter.c	Mon Aug 25 14:25:19 2003
>@@ -645,6 +645,7 @@
> 		/* Drop old route. */
> 		dst_release((*pskb)->dst);
> 		(*pskb)->dst = &rt->u.dst;
>+		iph->saddr = rt->rt_src;
> 	} else {
> 		/* non-local src, find valid iif to satisfy
> 		 * rp-filter when calling ip_route_input. */
>  
>

Re: policy routing on locally generated packets

[lartc]

Hi Again patrick,


On Mon, 2003-08-25 at 17:13, Patrick McHardy wrote:
> I don't know what i was thinking, this patch is going to confuse
> all kinds of stuff (especially conntrack & nat). Sorry.
> Patrick

Please don't be sorry -- your insight is so valuable.

I had a suspician that performing a full lookup afterward could create
more problems than it would solve.

So, I suppose (and humbly suggest) that the logical extension of this
thread would be that netfilter allow control over the output routing
process itself, thereby granting the administrator a means of
influencing route selection, src address, interface, etc.

iptables --append OUTROUTE ...

This would provide a way to fiddle with the packet properties without
the neccesity of changing bind() in a specific application ... or would
it? 

Sleep Well.

Charles

Re: policy routing on locally generated packets

[Patrick Schaaf]

> This would provide a way to fiddle with the packet properties without
> the neccesity of changing bind() in a specific application ... or would
> it? 

Regarding this path, two comments:

1) you can most easily LD_PRELOAD a small bit of library which replaces
bind() and friends, and does the thing you want.
2) the chbind functionality of http://www.solucorp.qc.ca/miscprj/s_context.hc
is a somewhat more / somewhat less intrusive method.

What about mangle/PREROUTING? Doesn't it / shouldn't it sit before a
routing decision for locally generated packets?

best regards
  Patrick

Re: policy routing on locally generated packets

[lartc]

Hi Patrick(s),


On Mon, 2003-08-25 at 21:07, Patrick Schaaf wrote:
> > This would provide a way to fiddle with the packet properties without
> > the neccesity of changing bind() in a specific application ... or would
> > it? 
> 
> Regarding this path, two comments:
> 
> 1) you can most easily LD_PRELOAD a small bit of library which replaces
> bind() and friends, and does the thing you want.
Most likely true, but I am a non-programmmer, and if I had preference,
changes to routing would be done without touching code.

> 2) the chbind functionality of http://www.solucorp.qc.ca/miscprj/s_context.hc
> is a somewhat more / somewhat less intrusive method.
I'll have a look, but (1)

> What about mangle/PREROUTING? Doesn't it / shouldn't it sit before a
> routing decision for locally generated packets?
PREROUTING is "before" a routing decision. but only affects packets
coming in over the wire, and not locally generated. 

My issue arises from ip accounting -- that is -- each user on a linux
box must have their wire traffic accounted for. As in the first message
of the thread:

#!/bin/bash
iptables --append OUTPUT --table mangle --match owner \
     --gid-owner 500 --jump MARK --set-mark 0x2

iptables --append OUTPUT --table mangle --match owner \
     --gid-owner 501 --jump MARK --set-mark 0x3

ip rule add fwmark 0x2 table 2
ip route add default dev eth0 src 192.168.0.1 via 192.168.0.254 table 2
ip rule add fwmark 0x3 table 3
ip route add default dev eth0 src 192.168.0.2 via 192.168.0.253 table 3
ip route flush cache
#end_script

This script would cause group 500 to use src address .1 and group 501
src address .2. Because each group has its own src address, I can do
wire level accounting/routing by group (or eventually user). In the
example, I am also selecting a router based on the gid. 

So I am searching for a hopefully simple way to implement this without
having to recompile a bunch of stuff.

Thanks Again!

Ciao

Charles




> 
> best regards
>   Patrick

Re: policy routing on locally generated packets

[Patrick Schaaf]

Hi Charles,

> > What about mangle/PREROUTING? Doesn't it / shouldn't it sit before a
> > routing decision for locally generated packets?
> PREROUTING is "before" a routing decision. but only affects packets
> coming in over the wire, and not locally generated. 

To me, it could/would be the most logical extension to make PREROUTING
also see outbound locally generated packets, before the routing step.

What do others think?

[snip per-uid accounting requirement]

> So I am searching for a hopefully simple way to implement this without
> having to recompile a bunch of stuff.

Your application would be far better served, IMHO, by a suitable, small
piece of code that does exactly what you need, instead of invoking complex
configuration all over the place.

First, the kernel already has a nice per-uid data structure, the
'struct user_struct'. It is used today to count the number of
processes a uid posesses. It could certainly be easily extended
with some counters. Second, those counters would get incremented
whenever a socket-using user level process sends or receives something.
Third, add some /proc/ file to read out the current counters, and a way
to reset the counters to 0.

Anybody here who A) needs the feature and B) has the skill to write the code?
If I'm not mistaken, it's 1-2 days of work.

best regards
  Patrick

Re: policy routing on locally generated packets

[lartc]

Hi Patrick,


On Tue, 2003-08-26 at 13:09, Patrick Schaaf wrote:
> Hi Charles,
> 
> > > What about mangle/PREROUTING? Doesn't it / shouldn't it sit before a
> > > routing decision for locally generated packets?
> > PREROUTING is "before" a routing decision. but only affects packets
> > coming in over the wire, and not locally generated. 
> 
> To me, it could/would be the most logical extension to make PREROUTING
> also see outbound locally generated packets, before the routing step.
> 
> What do others think?
I think it would be unfeasable to have locally generated packets be
associated with PREROUTING -- PREROUTING comes well before INPUT and
before QOS INGRESS eliminating it as a possible candidate.

Opening up OUTPUT ROUTING seems be the clearest path to allowing
administrative control to locally generated packets. 


> 
> [snip per-uid accounting requirement]
> 
> > So I am searching for a hopefully simple way to implement this without
> > having to recompile a bunch of stuff.
> 
> Your application would be far better served, IMHO, by a suitable, small
> piece of code that does exactly what you need, instead of invoking complex
> configuration all over the place.
> 
> First, the kernel already has a nice per-uid data structure, the
> 'struct user_struct'. It is used today to count the number of
> processes a uid posesses. It could certainly be easily extended
> with some counters. Second, those counters would get incremented
> whenever a socket-using user level process sends or receives something.
> Third, add some /proc/ file to read out the current counters, and a way
> to reset the counters to 0.
My example of traffic accounting is a very simple one -- I have seen
other requests on lartc to mangle packets during the OUTPUT ROUTING
process -- doing so makes it application independant. I personally
believe that it would be of high benefit, but of course, I am often
wrong!

Ciao

Charles

Re: policy routing on locally generated packets

[Cedric de Launois]

Le mar 26/08/2003 à 14:01, lartc@manchotnetworks.net a écrit :
> My example of traffic accounting is a very simple one -- I have seen
> other requests on lartc to mangle packets during the OUTPUT ROUTING
> process -- doing so makes it application independant. I personally
> believe that it would be of high benefit, but of course, I am often
> wrong!

Did you try the ROUTE target ?

Cedric

Re: policy routing on locally generated packets

[lartc]

Bonsoir Cedric,

Yes I know about your patch and actually reccomended it the other day.
It was in fact the the thread concerning routing policly on locally
generated packets that got me started with this thread. It has been a
subject on lartc from time to time, and I thought I might take a stab at
getting it solved.

Although I'm not a programmer, I have a feeling that the OUTPUT ROUTING
process was left out of netfilter for a good reason, although I'm not
adept enough to specifically identify why.

I'm on the road tomorrow, but in the evning, I'll do another post that
includes a schema that will hopefully clarify the issue.

Meilleures Salutations

Charles


On Tue, 2003-08-26 at 14:14, Cedric de Launois wrote:
> Le mar 26/08/2003 à 14:01, lartc@manchotnetworks.net a écrit :
> > My example of traffic accounting is a very simple one -- I have seen
> > other requests on lartc to mangle packets during the OUTPUT ROUTING
> > process -- doing so makes it application independant. I personally
> > believe that it would be of high benefit, but of course, I am often
> > wrong!
> 
> Did you try the ROUTE target ?
> 
> Cedric
> 
> 
> 

Re: policy routing on locally generated packets

[Henrik Nordstrom]

On Tue, 26 Aug 2003, Patrick Schaaf wrote:

> To me, it could/would be the most logical extension to make PREROUTING
> also see outbound locally generated packets, before the routing step.

Problem is that the packet does not really exists yet.. the routing which 
assigns source address is done on the socket, not on a packet. This 
special routing only occurs once per unbound socket.

Exception: This routing occurs on each UDP packet with unspecified source 
address, but again the routing is done separately from the actual packet I 
think..

Regards
Henrik

Re: policy routing on locally generated packets

[Patrick McHardy]

lartc@manchotnetworks.net wrote:

>My issue arises from ip accounting -- that is -- each user on a linux
>box must have their wire traffic accounted for. As in the first message
>of the thread:
>

Why don't add routes without specifing a source and nat the packets
to the prefered address ? Also, for accounting purposes the byte counters
of the matching rule should be fine i would think .. ?

Bye
Patrick

Re: policy routing on locally generated packets

[Henrik Nordstrom]

On 26 Aug 2003, lartc@manchotnetworks.net wrote:

> So I am searching for a hopefully simple way to implement this without
> having to recompile a bunch of stuff.

NAT of local connections can be of help here I think.

Also, there was some discussion regarding marking and rerouting of local 
packets some year ago. It was supposed to be fixed somehow (by NAT I 
think..)

Regards
Henrik

Re: policy routing on locally generated packets

[Patrick McHardy]

Patrick Schaaf wrote:

>What about mangle/PREROUTING? Doesn't it / shouldn't it sit before a
>routing decision for locally generated packets?
>

No locally generated packets don't pass the PRE_ROUTING hook. There
is no way to mangle these packets before routing.

>best regards
>  Patrick
>

dito ;)

Re: policy routing on locally generated packets

[Henrik Nordstrom]

On Tue, 26 Aug 2003, Patrick McHardy wrote:

> No locally generated packets don't pass the PRE_ROUTING hook. There
> is no way to mangle these packets before routing.

Indeed, but mangle OUTPUT reroutes the packets where needed 
(change in source/destination/mark/tos)

Regards
Henrik

Re: policy routing on locally generated packets [summary]

[lartc]

hi all, 

well, here's a stab at the summary of this thread, and hopefully a
salient solution:

issues arising from the following situation: 

[probably too big a diagram for the actual samples :-)]

             +------------------------------+ 
lan_a --->   | eth0        alice       eth2 | ----> isp_a 
             |                              | 
lab_b --->   | eth1                    eth3 | ----> isp_b 
             +------------------------------+ 

here's alice's ifconfig 
eth0   191.168.0.254/24 
eth1   192.168.1.254/24 
eth1:1 192.168.1.100/24
eth2   64.12.187.24/8 
eth2:1 64.12.187.25/8 
eth3   149.174.32.7/16 
eth3:1 149.174.32.8/16 

here's alice's default routing table: 
192.168.0.0/24 dev eth0  scope link 
192.168.1.0/24 dev eth1  scope link 
64.12.187.24/8 dev eth2  scope link 
149.174.32.7/16 dev eth3 scope link 
127.0.0.0/8 dev lo  scope link 


we would like that all applications (and their protocols) running under
userid 500 is routed to isp_b: 
 
(A)
iptables --append OUTPUT --table mangle --match owner \ 
         --uid-owner 500 --jump MARK --set-mark 0x2 
ip route add default dev eth3 src 149.174.32.7 table 2 
ip route flush cache 


conclusion: (A) doesn't work because OUTPUT ROUTING did a route lookup
without repsect to the fwmark -- lookup happend before the marking.

(B) 
ip route delete 192.168.1.0/24 dev eth1
ip route add 192.168.1.0/24 dev eth1 src 192.168.1.100
ip route flush cache

conclusion: (B) uses ip src address 192.168.1.100 for all
tcp/ip/udp/icmp communication. OUTPUT ROUTING used the src address
specified in the route.


final conclusion:
to effectively obtain policy routing on locally generated packets, the
OUTPUT ROUTING process would need to be hooked into netfilter thereby
letting packets be marked. OUTPUT ROUTING would then use the mark to
find a routing table via a routing rule. OUTPUT ROUTING would get the
route key fields using the "best" destination route within the routing
table, or use the default routing table if no suitable route was found.
(Cedric -- this would include the iif key therefore obviating/including
your patch through ip route commands)


iptables --append OUTROUTE --table mangle (expressions) \
         --jump MARK --set-mark 0x2


fwmark allows for 32 bits of precision, and it would be hard to imagine
that one would need more than that to implement a routing policy;
therfore, only table mangle need be present (imho) -- it offers plenty
of granularity.

resource issues -- since the fwmark is already part of the route key, it
can be placed in cache as usual.

thoughts, comments, ??

many thanks for all your help.


charles

Re: policy routing on locally generated packets [summary]

[Patrick McHardy]

lartc@manchotnetworks.net wrote:

>final conclusion:
>to effectively obtain policy routing on locally generated packets, the
>OUTPUT ROUTING process would need to be hooked into netfilter thereby
>letting packets be marked. OUTPUT ROUTING would then use the mark to
>find a routing table via a routing rule. OUTPUT ROUTING would get the
>route key fields using the "best" destination route within the routing
>table, or use the default routing table if no suitable route was found.
>(Cedric -- this would include the iif key therefore obviating/including
>your patch through ip route commands)
>
>
>iptables --append OUTROUTE --table mangle (expressions) \
>         --jump MARK --set-mark 0x2
>

It is not possible for two reasons:
- before output routing the packet doesn't have a source ip so it
would be weird to pass it through iptables tables.
- the source is chosen before the ip header is prepended, this
makes it impossible to pass it to iptables first.

Why can't you use NAT ?

Best regards,
Patrick

Re: policy routing on locally generated packets [summary]

[lartc]

Hi Patrick and Henrick,


On Wed, 2003-08-27 at 16:57, Patrick McHardy wrote:
> lartc@manchotnetworks.net wrote:
> 
> >final conclusion:
> >to effectively obtain policy routing on locally generated packets, the
> >OUTPUT ROUTING process would need to be hooked into netfilter thereby
> >letting packets be marked. OUTPUT ROUTING would then use the mark to
> >find a routing table via a routing rule. OUTPUT ROUTING would get the
> >route key fields using the "best" destination route within the routing
> >table, or use the default routing table if no suitable route was found.
> >(Cedric -- this would include the iif key therefore obviating/including
> >your patch through ip route commands)
> >
> >
> >iptables --append OUTROUTE --table mangle (expressions) \
> >         --jump MARK --set-mark 0x2
> >
> 
> It is not possible for two reasons:
> - before output routing the packet doesn't have a source ip so it
> would be weird to pass it through iptables tables.
actually, thats just what i want to do -- iptables would mark the packet
only and the output routing would respect the mark ... but 

> - the source is chosen before the ip header is prepended, this
> makes it impossible to pass it to iptables first.
this pretty much says it all ... oh well :-/ 


> Why can't you use NAT ?
i tried the following

iptables --append OUTPUT --table mangle --match owner --uid-owner 500
--jump MARK --set-mark 0x2

iptables --append POSTROUTING --table nat --match mark \
         --mark 0x2 --jump SNAT --to-source 192.168.1.100
ip route flush cache


*tcp/udp work
*icmp and arp (yeah, L2) dont -- but they do when the src address is set
by ip route 
*ip route lets you define all the nitty gritty (key fields) in one place
in a nice garden setting ...

ciao

charles

Re: policy routing on locally generated packets [summary]

[Henrik Nordstrom]

ons 2003-08-27 klockan 16.46 skrev lartc@manchotnetworks.net:

> final conclusion:
> to effectively obtain policy routing on locally generated packets, the
> OUTPUT ROUTING process would need to be hooked into netfilter thereby
> letting packets be marked. OUTPUT ROUTING would then use the mark to
> find a routing table via a routing rule.

But it is... The OUTPUT mangle chain is before routing. The exception is
source address assignment on unbound sockets which is part of building
the packet to send. The source IP address assignment is not really a
routing decision but uses the same engine in the kernel.

If you mangle OUTPUT packets in iptables then the packet will be
rerouted if already routed. But the source IP address is already
assigned and will not be automatically changed unless you instruct
iptables to do so via NAT.

Regards
Henrik

-- 
Henrik Nordstrom <hno@marasystems.com>
MARA Systems AB

Re: policy routing on locally generated packets

[Patrick McHardy]

Hi Charles,

lartc@manchotnetworks.net wrote:

> So, I suppose (and humbly suggest) that the logical extension of this
>
>thread would be that netfilter allow control over the output routing
>process itself, thereby granting the administrator a means of
>influencing route selection, src address, interface, etc.
>
>iptables --append OUTROUTE ...
>

There i a ROUTE target, maybe that helps. Before i can discuss this
further i have to do some testing myself, my first attempt to route
by netfilter mark wasn't successful at all, so please give me a day
or two.

Bye,
Patrick