Multipath routing in Linux 2.6

Multipath routing in Linux 2.6

[Tore Anderson]

   Hello list,

   I've been trying to figure out how to make equal-cost multipath
  routing work, with no luck.  Asked on the LARTC list with no success,
  and attempts to contact the two authors privately yielded one bounce
  while the other declined to answer in private and pointed me to this
  list.  So I'll just include the rest of the mail I sent them here
  (after doing a search-and-replace on the first three octets of all
  addresses, I'm a bit paranoid), hopefully someone has some suggestions
  for me...

   I'm using 2.6.20 on an x86_64 machine.  I'm adding my route thusly:

     ip route add table 100 default \
       nexthop via 1.1.1.1 nexthop via 1.1.1.9

   It shows correctly up in the routing table:

     root@router:~# ip route show table 100
     default
             nexthop via 1.1.1.1  dev vlan11 weight 1
             nexthop via 1.1.1.9  dev vlan12 weight 1
     [...]

   I'm sending traffic from a relatively busy network to this table:

     root@router:~# ip rule
     [...]
     21000:  from 1.1.2.128/26 lookup 100
     [...]

   I can verify with tcpdump that the rule works correctly and that the
  route is used.  However, the traffic is without exception routed via
  1.1.1.9, not a single packet is sent to 1.1.1.1.  If I however swap
  the two nexthops while adding the route, all traffic is sent to
  1.1.1.1, and nothing ends up at 1.1.1.9.

   I've tried loading and unloading the multipath_{wrandom,rr,random,drr}
  modules, removing and readding the route, and flushing the routing
  cache.  Several times and in different order.  Nothing affects the
  behaviour though, all of the traffic is sent to the router specified as
  the second nexthop on the "ip route add" command line.

   I feel I'm missing something essential here but I have no idea what.
  Google only tells me about others having roughly the same problem but
  never any solution.  Do you have any suggestions for me?  If I can make
  this work I will be happy to document how and try to have that included
  in the next kernel/iproute release and hopefully nobody will bother you
  about it again.

   Thanks for your time!

Kind regards
-- 
Tore Anderson

Re: Multipath routing in Linux 2.6

[Jarek Poplawski]

On 06-03-2007 21:36, Tore Anderson wrote:
> 
>   Hello list,
> 
>   I've been trying to figure out how to make equal-cost multipath
>  routing work, with no luck.  Asked on the LARTC list with no success,

It is probably one of the most often asked questions
on the LARTC, so I'd suggest to look at its archives.

>   I'm sending traffic from a relatively busy network to this table:
...
>   I've tried loading and unloading the multipath_{wrandom,rr,random,drr}

Multipath with caching doesn't work with forwarding.
...
>   I feel I'm missing something essential here but I have no idea what.
>  Google only tells me about others having roughly the same problem but
>  never any solution. [...]

It must be this fake google.

Some wrandom suggestions:

CONFIG_IP_ROUTE_MULTIPATH = y
CONFIG_IP_ROUTE_MULTIPATH_CACHED = n
rp_filter turned off
iptables CONNMARK or Julian Anastasov's patch
more ip route ... & ip rule ...
go to the LARTC again with: why still doesn't work...

Regards,
Jarek P.

[PATCH] Improve cached ECMP documentation

[Tore Anderson]

Document that equal-cost multipath routing with caching does not work
for forwarded packets.

Signed-Off-By: Tore Anderson <tore@linpro.no>
---

* Jarek Poplawski

> It is probably one of the most often asked questions
> on the LARTC, so I'd suggest to look at its archives.
>
> [...]
> 
> Multipath with caching doesn't work with forwarding.

  Well, I did look at LARTC's archives (and again after reading your
 mail), still couldn't find anything.  Sorry for bothering you, though.
 When I disabled the support it worked just fine (no need for iptables
 CONNMARK trickery or any of Julian's patches), so thanks for your help!

  Anyway, it strikes me as a bit odd that this feature is found under
 CONFIG_IP_ADVANCED_ROUTER when it doesn't work for routers.  That
 could at least be better documented - here's a patch to do just that.

--- a/net/ipv4/Kconfig
+++ b/net/ipv4/Kconfig
@@ -124,6 +124,10 @@ config IP_ROUTE_MULTIPATH_CACHED
 	  routing cache. If you say Y here, alternative routes are cached
 	  and on cache lookup a route is chosen in a configurable fashion.
 
+	  Be warned that this feature does not work for forwarded packets,
+	  and that enabling it will break equal cost multipath routing for
+	  forwarded traffic.
+
 	  If unsure, say N.
 
 config IP_ROUTE_MULTIPATH_RR

Re: [PATCH] Improve cached ECMP documentation

[Jarek Poplawski]

On Wed, Mar 07, 2007 at 01:37:44PM +0100, Tore Anderson wrote:
> Document that equal-cost multipath routing with caching does not work
> for forwarded packets.
> 
> Signed-Off-By: Tore Anderson <tore@linpro.no>
> ---
> 
> * Jarek Poplawski
> 
> >It is probably one of the most often asked questions
> >on the LARTC, so I'd suggest to look at its archives.
> >
> >[...]
> >
> >Multipath with caching doesn't work with forwarding.
> 
>  Well, I did look at LARTC's archives (and again after reading your
> mail), still couldn't find anything.  Sorry for bothering you, though.

I don't feel bothered at all. Your letter could
make an impression there is a lack of help and
google not working, but it's not always true. 
So, sometimes people probably wait to see, they
will only help and not do all your work. 

> When I disabled the support it worked just fine (no need for iptables
> CONNMARK trickery or any of Julian's patches), so thanks for your help!

Without this there is a possibility the route could be
changed during connection.

>  Anyway, it strikes me as a bit odd that this feature is found under
> CONFIG_IP_ADVANCED_ROUTER when it doesn't work for routers.  That
> could at least be better documented - here's a patch to do just that.

Maybe it's intended for something even more advanced,
I don't know. I checked this long time ago and later
seen someone was trying to fix something around this
- so I don't know what is the current state. But you
are right - this should be better documented.

Cheers,
Jarek P.

Re: [PATCH] Improve cached ECMP documentation

[Tore Anderson]

* Jarek Poplawski

> I don't feel bothered at all. Your letter could
> make an impression there is a lack of help and
> google not working, but it's not always true. 
> So, sometimes people probably wait to see, they
> will only help and not do all your work.

   Of course.  I tried hard to figure it out on my own, but couldn't.
  I'm sorry if I was unable to communicate that properly - I never would
  ask someone for help without having attempted to figure it out on my
  own first.

> Without this there is a possibility the route could be
> changed during connection.

   That's no problem, it could be changed between each packet for all I
  care.  Connections are a TCP concept, and routers normally only care
  about IPs.  The routes change all the time anyway, multipath or not.
  I'm quite certain I get way more than one BGP update per second in a
  normal day at least.

   So what I'm trying to achieve is only to better balance outgoing
  traffic between two transit links to avoid saturating the physical
  layer on one of the links.  My multipath route achieves this nicely
  when CONFIG_IP_ROUTE_MULTIPATH_CACHED is unset - thanks again!  It
  behaved just like a normal singlepath route when it was enabled though
  (even with no multipath modules loaded) - not quite what I expected.

Regards
-- 
Tore Anderson