suggestion for routing code improvement

suggestion for routing code improvement

[Chris Friesen]

Well, we're into a new experimental branch, so I figured the time had come to
bring back an idea that didn't make it in a year ago.

Currently, if you add custom routes and then down the device that the routes are
associated with, the routes are then removed.  This is entirely understandable.

However, when you then up the interface, the routes do not come back up
automatically.  This is the part that I don't agree with.

Addresses, automatic routes, and custom rules either stay in effect or are
brought back up by the system when an interface is downed and then upped.  The
only exception to this is custom routes.

I propose that these custom routes be stored in some fashion, and then be
brought back up when the device is brought back up.  This would make certain
types of link management much simpler.  Custom routes, once added, would stay in
effect until explicitly removed.  As it stands now, any time you down/up a link
you have to then go back and add all the custom routes again.  This can be a
pain in the butt.

So, what do you guys think?  Is this a reasonable thing to do?  I think that it
makes the system nicely symmetrical, as opposed to the asymmetrical handling of
current kernels.

I don't have a really good grasp of what data structures are avialable to us in
the routing code, so could someone with a better understanding of the code
comment as to how hard this would be to implement?

Thanks,


Chris




-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: suggestion for routing code improvement

[Andi Kleen]

On Wed, Apr 10, 2002 at 01:51:12PM -0400, Chris Friesen wrote:
> 
> Well, we're into a new experimental branch, so I figured the time had come to
> bring back an idea that didn't make it in a year ago.
> 
> Currently, if you add custom routes and then down the device that the routes are
> associated with, the routes are then removed.  This is entirely understandable.
> 
> However, when you then up the interface, the routes do not come back up
> automatically.  This is the part that I don't agree with.
> 
> Addresses, automatic routes, and custom rules either stay in effect or are
> brought back up by the system when an interface is downed and then upped.  The
> only exception to this is custom routes.
> 
> I propose that these custom routes be stored in some fashion, and then be
> brought back up when the device is brought back up.  This would make certain
> types of link management much simpler.  Custom routes, once added, would stay in
> effect until explicitly removed.  As it stands now, any time you down/up a link
> you have to then go back and add all the custom routes again.  This can be a
> pain in the butt.
> 
> So, what do you guys think?  Is this a reasonable thing to do?  I think that it
> makes the system nicely symmetrical, as opposed to the asymmetrical handling of
> current kernels.
> 
> I don't have a really good grasp of what data structures are avialable to us in
> the routing code, so could someone with a better understanding of the code
> comment as to how hard this would be to implement?


I don't think it would be very hard. Internally device up/down state 
and the IP address lists and IP state are separated. The routes and addresses
are currently removed triggered by the NETDEV_DOWN event on the netdev_chain.
There is no fundamental reason AFAIK that needs to be done, you could
not react to the DOWN event in IP in some circumstances. the upper layer
should handle down devices already because that can always happen even now
due to races at shutdown.


-Andi

suggestion for routing code improvement

[Robert Olsson]

Chris Friesen writes:
 > So, what do you guys think?  Is this a reasonable thing to do?  I think that it
 > makes the system nicely symmetrical, as opposed to the asymmetrical handling of
 > current kernels.

 Hello!

 Why not leave the routing policy job to a routing daemon?

 There is risk of routing havoc if the kernel start acting as one.

  L-uu1:/# ip route list | wc -l 
   110441

 This Linux box has 110441 bgp routes. Internet routing is very much
 like a living organism. Routes comes and goes.

 If some interface goes down for some reason the routing daemon gets it's 
 netlink message "link down" and recalcs the routing topologi.

 If the interface comes back the router daemon recalcs again and installs
 appropriate routes for this moment which may very well be different 
 compared to before "link down".

 Cheers.

						--ro

Re: suggestion for routing code improvement

[Chris Friesen]

Robert Olsson wrote:
> 
> Chris Friesen writes:
>  > So, what do you guys think?  Is this a reasonable thing to do?  I think that it
>  > makes the system nicely symmetrical, as opposed to the asymmetrical handling of
>  > current kernels.
> 
>  Hello!
> 
>  Why not leave the routing policy job to a routing daemon?

Because I've got static routes, and I know exactly what they are.

>  This Linux box has 110441 bgp routes. Internet routing is very much
>  like a living organism. Routes comes and goes.

The box(es) which inspired this sit on a private network, and once they are
brought into service, the routes never change.  However, there is a command from
our gui to manually drop and raise the ethernet link (just in case something
goes wrong and can't be handled automatically) and it would simplify our code
greatly if the routes that are automatically deleted would be automatically put
back.

>  If the interface comes back the router daemon recalcs again and installs
>  appropriate routes for this moment which may very well be different
>  compared to before "link down".

In this case, my software *is* essentially the routing daemon, and I want it to
be simpler to maintain.

I think the concept is simple: I added some routes, and I think they should stay
there until I remove them or the machine reboots.  Doesn't this seem like a
logical behaviour?

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: suggestion for routing code improvement

[Robert Olsson]

Chris Friesen writes:
 > The box(es) which inspired this sit on a private network, and once they are
 > brought into service, the routes never change.  However, there is a command from
 > our gui to manually drop and raise the ethernet link (just in case something
 > goes wrong and can't be handled automatically) and it would simplify our code
 > greatly if the routes that are automatically deleted would be automatically put
 > back.

 If we limit us to "static routes" for the other routes we must definitely leave to 
 some with routing/topologi knowledge and we must not break systems with "routing 
 daemons".

 If we look into the routing world a "route" can come from different origins/routing
 protocols static/ospf/rip/bgp and there is preferences to decide which one to 
 install.

 So

 ip route add x y proto static 

 Is registered by routing daemon and installed if it has the best pref. and the 
 route is hidden otherwise to be restored when appropriate. And the route is invalid 
 but kept by the daemon over "link downs" 

 And I think

 ip route rep x y proto static 

 or something similar at link up for restoring "static" routes would not break 
 for the daemons either. But I got a feeling it can hurt more than it helps.


 > In this case, my software *is* essentially the routing daemon, and I want it to
 > be simpler to maintain.
 
 Well router daemons belongs to "user space" and via netlink you should be able to 
 do most of what you want.

 
 Cheers.

							--ro

 

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Thu, 11 Apr 2002, Robert Olsson wrote:

>  If we limit us to "static routes" for the other routes we must definitely leave to 
>  some with routing/topologi knowledge and we must not break systems with "routing 
>  daemons".

	include/linux/rtnetlink.h already contains the needed RTPROT_xxx
definitions. The most used daemons don't use RTPROT_STATIC. The kernel
does not know that the daemon registers static routes, they all
have its own RTPROT_value. The static routes are marked as such only
in the daemon's config file. May be it is possible value RTPROT_STATIC
to be marked in comments as a kernel property.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[Robert Olsson]

Julian Anastasov writes:
 > 
 > 	include/linux/rtnetlink.h already contains the needed RTPROT_xxx
 > definitions. The most used daemons don't use RTPROT_STATIC. The kernel
 > does not know that the daemon registers static routes, they all
 > have its own RTPROT_value. The static routes are marked as such only
 > in the daemon's config file. May be it is possible value RTPROT_STATIC
 > to be marked in comments as a kernel property.

Hello!

There is already RTPROT_KERNEL and proto RTPROT_STATIC is way for the
administrator to interact with the routing daemon even if is a you say
that this is not currently implemented by all daemons. I have used this 
with gated.

IMHO even static routes implements a routing policy from some administrator
view and should therefore by configured, debugged etc. This job has to done 
somewhere and a userland daemon seems most adequate at least in my eyes.

And with the current model the "network" comes up in a clean fashion. I think 
it easier to just add the needed routes rather to check the old history and 
deal with this. Probably we need to check is address, netmask changes etc
so the old routes will be installed in the same context.

A very simple daemon for static routes would do the job we are talking about 
here and should be easy to configure through config files or just via netlink. 
And just replace it with zebra/gated/bird when the complexity requires it.

Cheers.

						--ro

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Thu, 11 Apr 2002, Robert Olsson wrote:

> There is already RTPROT_KERNEL and proto RTPROT_STATIC is way for the
> administrator to interact with the routing daemon even if is a you say
> that this is not currently implemented by all daemons. I have used this
> with gated.

	Well, I now see, it is used in gated. But only in one route table
which is a drawback.

> IMHO even static routes implements a routing policy from some administrator
> view and should therefore by configured, debugged etc. This job has to done
> somewhere and a userland daemon seems most adequate at least in my eyes.

	Agreed. Add to this that sometimes many such agents play with
the routes but in its own area (different tables, etc) and the things
become more complex. I still don't know for daemon that can work with
multiple routing tables in Linux, some even don't recognize the
preferred source IPs and the ip rules.

> And with the current model the "network" comes up in a clean fashion. I think
> it easier to just add the needed routes rather to check the old history and
> deal with this. Probably we need to check is address, netmask changes etc
> so the old routes will be installed in the same context.
>
> A very simple daemon for static routes would do the job we are talking about
> here and should be easy to configure through config files or just via netlink.
> And just replace it with zebra/gated/bird when the complexity requires it.

	More correctly, the settings are more complex than the mentioned
daemons can handle. Sticking with one route table is not enough in most
of the cases. This is the main reason the mentioned patches for static
routes to exist. They are more manageable (with scripts) for setups where
routing protocols are not used.

> Cheers.
>
> 						--ro

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[jamal]

On Thu, 11 Apr 2002, Julian Anastasov wrote:

>
> 	Hello,
>
> On Thu, 11 Apr 2002, Robert Olsson wrote:
>
> > There is already RTPROT_KERNEL and proto RTPROT_STATIC is way for the
> > administrator to interact with the routing daemon even if is a you say
> > that this is not currently implemented by all daemons. I have used this
> > with gated.
>
> 	Well, I now see, it is used in gated. But only in one route table
> which is a drawback.

I thought gated was capable of using at least main and local.
If i am not mistaken zebra is now capable of using more tables as well.
It would probably be actually better policy to enter all static routes in
one table.
Robert, when you enter a static route from gated, is it registered as
proto gated or proto static?
While i like Julians patch (adding no complexity, IMO) I see that the
functionality could be very easily moved outside the kernel where you
could also do a lot more fancy things (very complex decision making
example: based on which devs go down, what next multihops to use etc).
Unfortunately, it does require extra code in user space.
The question is: Would people who need this functionality be not very lazy
and actually fetch this code and compile it? ;->

cheers,
jamal

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Thu, 11 Apr 2002, jamal wrote:

> > 	Well, I now see, it is used in gated. But only in one route table
> > which is a drawback.
>
> I thought gated was capable of using at least main and local.

	Looking in the sources, main+default and after patching
with one table more

> If i am not mistaken zebra is now capable of using more tables as well.

	Last time I checked it 6 months ago and today I don't see
much progress on the web site but you can be right :)

> It would probably be actually better policy to enter all static routes in
> one table.

	It is not always possible, all we are using Advanced Linux
Networking, though :)

> Robert, when you enter a static route from gated, is it registered as
> proto gated or proto static?

	I see that gated uses many protos while Zebra uses small
number (one?). Actually, gated can use RTPROT_STATIC for its table.

> While i like Julians patch (adding no complexity, IMO) I see that the
> functionality could be very easily moved outside the kernel where you
> could also do a lot more fancy things (very complex decision making
> example: based on which devs go down, what next multihops to use etc).
> Unfortunately, it does require extra code in user space.

	Yes, we can ping indirect nexthops, do layer 7 health checks,
alter the nexthops' weights. Usually we don't demand such super
job from a routing daemon, the things are mostly "do it yourself" :)

> cheers,
> jamal

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[jamal]

On Thu, 11 Apr 2002, Julian Anastasov wrote:

>
> 	Hello,
>
> > If i am not mistaken zebra is now capable of using more tables as well.
>
> 	Last time I checked it 6 months ago and today I don't see
> much progress on the web site but you can be right :)
>

hrm. Maybe its in their commercial version ;-<
Note how slow they are in updating it these days. Someone approached
me a while back with suggestions to join in forking (and liberate) Zebra
from the current (lack of) activity. My answer at the time is to put more
effort in Bird probably.

> 	Yes, we can ping indirect nexthops, do layer 7 health checks,
> alter the nexthops' weights. Usually we don't demand such super
> job from a routing daemon, the things are mostly "do it yourself" :)
>

My sentiments exactly. Any such beast when written should be policy
driven. If i dont need layer 7 health checks as stimulus, then
dont use them. Other things would be remote SNMP queries etc.

cheers,
jamal

Re: suggestion for routing code improvement

[Robert Olsson]

Hello!

Julian Anastasov writes:
 > 
 > 	Well, I now see, it is used in gated. But only in one route table
 > which is a drawback.
 
 gated netlink code came from Alexey. I had to adjust the preference for 
 "static" in gated to get the desired effect.
 
 > 
 > 	More correctly, the settings are more complex than the mentioned
 > daemons can handle. Sticking with one route table is not enough in most
 > of the cases. This is the main reason the mentioned patches for static
 > routes to exist. They are more manageable (with scripts) for setups where
 > routing protocols are not used.

 Well we can be happy for the well-designed routing/netlink system Linux 
 got and as you say routing software does not yet fully utilize this. 
 But I think we agree that we shouldn't have kernel to compensate for this.

 I did some benchmarking/profiling on the routing lookup/cache/garbage 
 collection it seems to be well done too. Having listened to all fuzz 
 about routing lookups at GIGE speeds. :-)

 Cheers.

						--ro

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Thu, 11 Apr 2002, Robert Olsson wrote:

>  Well we can be happy for the well-designed routing/netlink system Linux
>  got and as you say routing software does not yet fully utilize this.
>  But I think we agree that we shouldn't have kernel to compensate for this.

	Agreed. We are not trying to insert routing daemon into kernel :)

>  I did some benchmarking/profiling on the routing lookup/cache/garbage
>  collection it seems to be well done too. Having listened to all fuzz
>  about routing lookups at GIGE speeds. :-)

	If you use another hash function for the route cache you can
even get better :) In my tests with many hosts i got 8% increase in
performance.

>  Cheers.
>
> 						--ro

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[Robert Olsson]

Julian Anastasov writes:

 > 	If you use another hash function for the route cache you can
 > even get better :) In my tests with many hosts i got 8% increase in
 > performance.

 Interesting. At what rate of new connections/s did you test?
 
 Cheers	.

						--ro

 

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Thu, 11 Apr 2002, Robert Olsson wrote:

>  > 	If you use another hash function for the route cache you can
>  > even get better :) In my tests with many hosts i got 8% increase in
>  > performance.
>
>  Interesting. At what rate of new connections/s did you test?

	It was on 100mbit, I already don't remember the exact tests
and number of clients, etc. The patch is archived ..., here it is.
You can try it yourself:


--- net/ipv4/route.c.orig	Wed Jun  6 17:51:49 2001
+++ net/ipv4/route.c	Wed Jun  6 18:25:45 2001
@@ -203,11 +203,9 @@

 static __inline__ unsigned rt_hash_code(u32 daddr, u32 saddr, u8 tos)
 {
-	unsigned hash = ((daddr & 0xF0F0F0F0) >> 4) |
-			((daddr & 0x0F0F0F0F) << 4);
-	hash ^= saddr ^ tos;
-	hash ^= (hash >> 16);
-	return (hash ^ (hash >> 8)) & rt_hash_mask;
+	unsigned hash = (saddr + daddr + tos);
+	hash = ntohl(hash) * 2654435761UL;
+	return hash & rt_hash_mask;
 }

 static int rt_cache_get_info(char *buffer, char **start, off_t offset,



I'm using attacking program with configurable number of client IPs:

http://www.linuxvirtualserver.org/~julian/#testlvs

	Sorry that it does not have flood rate limits, use QoS in
the flooding host :) You can try the patch with different tools,
of course.

>  Cheers	.
>
> 						--ro

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[Robert Olsson]

Julian Anastasov writes:
 > >  Interesting. At what rate of new connections/s did you test?
 > 
 > 	It was on 100mbit, I already don't remember the exact tests
 > and number of clients, etc. The patch is archived ..., here it is.
 > You can try it yourself:

 > http://www.linuxvirtualserver.org/~julian/#testlvs
 > 
 > 	Sorry that it does not have flood rate limits, use QoS in
 > the flooding host :) You can try the patch with different tools,
 > of course.

 Very intesting. Well seems like I don't get out of the lab until the
 snow comes....
 
 We're testing GIGE switches now. With one Linux box with five e1000
 we can put a background load of almost 5 Gbit/s into the switch.
 And check latencies through other pair of switch ports. 

 Cheers.

						--ro

Re: suggestion for routing code improvement

[Julian Anastasov]

	Hello,

On Wed, 10 Apr 2002, Chris Friesen wrote:

> However, when you then up the interface, the routes do not come back up
> automatically.  This is the part that I don't agree with.
>
> Addresses, automatic routes, and custom rules either stay in effect or are
> brought back up by the system when an interface is downed and then upped.  The
> only exception to this is custom routes.

	Check this implementation of static routes (new version
is coming soon...):

http://www.linuxvirtualserver.org/~julian/#routes

The exact patches for static routes:
http://www.linuxvirtualserver.org/~julian/00_static_routes-2.4.16-6.diff
http://www.linuxvirtualserver.org/~julian/00_static_routes-2.2.20-6.diff

> So, what do you guys think?  Is this a reasonable thing to do?  I think that it
> makes the system nicely symmetrical, as opposed to the asymmetrical handling of
> current kernels.

	The patches are not perfect but I hope they can help you:

ip route add XXX proto static

	Such routes survive only device state changes. They are
removed when the last IP address is deleted or the device is
unregistered. Of course, it is recommended that the multipath
routes are recreated when a device used in nexthop is unregistered
or all its addresses are deleted. So, it needs some scripting and
help from user space.

> Thanks,
>
>
> Chris

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: suggestion for routing code improvement

[Nivedita Singhvi]

 > > So, what do you guys think?  Is this a reasonable thing
 > > to do?  I think that it
 > > makes the system nicely symmetrical, as opposed to the
 > > asymmetrical handling of current kernels.

 > Hello!

 > Why not leave the routing policy job to a routing daemon?

 > There is risk of routing havoc if the kernel start acting
 > as one.

 > L-uu1:/# ip route list | wc -l
 > 110441

 > This Linux box has 110441 bgp routes. Internet routing is
 > very much like a living organism. Routes comes and goes.

 But is that the norm? Not all linux installations are large
 internet nodes..I'd say at least a significant minority of
 hosts typically have just a few static routes, not running
 a routing daemon, who would be helped by some mechanism to
 auto reinstall routes..(perhaps a generously configurable
 one)

thanks,
Nivedita