Bug w/ (policy) routing

Bug w/ (policy) routing

[Olivier Brunel]

Hi,

(Please cc me as I'm not subscribed to the list, thanks.)

I'm trying to set things up using some policy routing, and having some
weird issues I can't really explain. It looks to me like there might be
a bug somewhere...

This is done under Arch Linux 64bits, iproute2 4.9.0 (`ip -V` says ip
utility, iproute2-ss161212), kernel 4.8.13

Basically here's what I could reduce it to:
- create a new network namespace, create a pair of veth devices: one in
there, one sent back to the original namespace
- I'm giving them IPs 10.4.0.1 (original namespace) & 10.4.0.2 (new
namespace)
- in that new namespace, I'm trying to add a route to 10.4.0.1, but
  inside a new table. I also want a default route via 10.4.0.1 on the
  table main. It seems to work, only not really...

It's not very easy to describe so hopefully this will make things
clearer:

$ sudo unshare -n sh
sh-4.4# ip rule add table 50 prio 50
sh-4.4# ip link add test type veth peer name test2
sh-4.4# ip addr add 10.4.0.2 dev test
sh-4.4# ip link set dev test up
sh-4.4# ip link set netns 1 dev test2
# back in original namespace, we add 10.4.0.1 to test2 and bring it up

sh-4.4# ip route add 10.4.0.1 dev test table 50
sh-4.4# ip route add default via 10.4.0.1 dev test
sh-4.4# ip route flush cache
sh-4.4# ip rule
0:	from all lookup local 
50:	from all lookup 50 
32766:	from all lookup main 
32767:	from all lookup default 
sh-4.4# ip route show table 50
10.4.0.1 dev test scope link 
sh-4.4# ip route get 10.4.0.1
10.4.0.1 via 10.4.0.1 dev test table local src 10.4.0.2 
    cache 
# !?? why isn't table 50 used as, I believe, it should. And why
does adding a rule "fixes" it :

sh-4.4# ip rule add prio 55555
sh-4.4# ip route get 10.4.0.1
10.4.0.1 dev test table 50 src 10.4.0.2 
    cache 
# deleting the new rule makes no difference. It could even have been
done right after adding it. It's like it just triggered something
(reload, cache flushed, ...)
As seen I did flush cached routes as mentionned in the man page, I don't
know of anything else that would need done to "refresh" things?

Any ideas as to why this is happening? Should this work as I expect it,
or is there anything I'm doing wrong?

Thanks,

Re: Bug w/ (policy) routing

[David Ahern]

On 12/30/16 4:00 PM, Olivier Brunel wrote:
> Hi,
> 
> (Please cc me as I'm not subscribed to the list, thanks.)
> 
> I'm trying to set things up using some policy routing, and having some
> weird issues I can't really explain. It looks to me like there might be
> a bug somewhere...
> 
> This is done under Arch Linux 64bits, iproute2 4.9.0 (`ip -V` says ip
> utility, iproute2-ss161212), kernel 4.8.13
> 
> Basically here's what I could reduce it to:
> - create a new network namespace, create a pair of veth devices: one in
> there, one sent back to the original namespace
> - I'm giving them IPs 10.4.0.1 (original namespace) & 10.4.0.2 (new
> namespace)
> - in that new namespace, I'm trying to add a route to 10.4.0.1, but
>   inside a new table. I also want a default route via 10.4.0.1 on the
>   table main. It seems to work, only not really...
> 
> It's not very easy to describe so hopefully this will make things
> clearer:
> 
> $ sudo unshare -n sh

The main and local fib tables start merged into a single fib_table instance.

> sh-4.4# ip rule add table 50 prio 50
> sh-4.4# ip link add test type veth peer name test2
> sh-4.4# ip addr add 10.4.0.2 dev test
> sh-4.4# ip link set dev test up
> sh-4.4# ip link set netns 1 dev test2
> # back in original namespace, we add 10.4.0.1 to test2 and bring it up
> 
> sh-4.4# ip route add 10.4.0.1 dev test table 50
> sh-4.4# ip route add default via 10.4.0.1 dev test
> sh-4.4# ip route flush cache
> sh-4.4# ip rule
> 0:	from all lookup local 
> 50:	from all lookup 50 
> 32766:	from all lookup main 
> 32767:	from all lookup default 
> sh-4.4# ip route show table 50
> 10.4.0.1 dev test scope link 
> sh-4.4# ip route get 10.4.0.1
> 10.4.0.1 via 10.4.0.1 dev test table local src 10.4.0.2 
>     cache 
> # !?? why isn't table 50 used as, I believe, it should. And why

The default rule set has the local table at priority 0 so all lookups hit that table first:

# ip ru ls
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

For some reason it hits a match doing the lookup in the merged local/main fib_table for this ip route get.

> does adding a rule "fixes" it :
> 
> sh-4.4# ip rule add prio 55555

Adding this rule causes the local and main tables to be split into actual separate fib tables. Doing so causes the lookup in the local table to fail, so the lookup continues to the next rule -- which is your prio 50 table 50 rule.

I did not look into why the earlier rule addition did not cause the unmerge to happen -- it should have.


> sh-4.4# ip route get 10.4.0.1
> 10.4.0.1 dev test table 50 src 10.4.0.2 
>     cache 
> # deleting the new rule makes no difference. It could even have been
> done right after adding it. It's like it just triggered something
> (reload, cache flushed, ...)
> As seen I did flush cached routes as mentionned in the man page, I don't
> know of anything else that would need done to "refresh" things?
> 
> Any ideas as to why this is happening? Should this work as I expect it,
> or is there anything I'm doing wrong?

For your purposes now this should fix the problem for you:

ip ru del from all lookup local
ip ru add prio 32765 from all lookup local

Re: Bug w/ (policy) routing

[Olivier Brunel]

On Sat, 31 Dec 2016 13:15:44 -0700
David Ahern <dsa@cumulusnetworks.com> wrote:

> On 12/30/16 4:00 PM, Olivier Brunel wrote:
> > Hi,
> > 
> > (Please cc me as I'm not subscribed to the list, thanks.)
> > 
> > I'm trying to set things up using some policy routing, and having
> > some weird issues I can't really explain. It looks to me like there
> > might be a bug somewhere...
> > 
> > This is done under Arch Linux 64bits, iproute2 4.9.0 (`ip -V` says
> > ip utility, iproute2-ss161212), kernel 4.8.13
> > 
> > Basically here's what I could reduce it to:
> > - create a new network namespace, create a pair of veth devices:
> > one in there, one sent back to the original namespace
> > - I'm giving them IPs 10.4.0.1 (original namespace) & 10.4.0.2 (new
> > namespace)
> > - in that new namespace, I'm trying to add a route to 10.4.0.1, but
> >   inside a new table. I also want a default route via 10.4.0.1 on
> > the table main. It seems to work, only not really...
> > 
> > It's not very easy to describe so hopefully this will make things
> > clearer:
> > 
> > $ sudo unshare -n sh  
> 
> The main and local fib tables start merged into a single fib_table
> instance.
> 
> > sh-4.4# ip rule add table 50 prio 50
> > sh-4.4# ip link add test type veth peer name test2
> > sh-4.4# ip addr add 10.4.0.2 dev test
> > sh-4.4# ip link set dev test up
> > sh-4.4# ip link set netns 1 dev test2
> > # back in original namespace, we add 10.4.0.1 to test2 and bring it
> > up
> > 
> > sh-4.4# ip route add 10.4.0.1 dev test table 50
> > sh-4.4# ip route add default via 10.4.0.1 dev test
> > sh-4.4# ip route flush cache
> > sh-4.4# ip rule
> > 0:	from all lookup local 
> > 50:	from all lookup 50 
> > 32766:	from all lookup main 
> > 32767:	from all lookup default 
> > sh-4.4# ip route show table 50
> > 10.4.0.1 dev test scope link 
> > sh-4.4# ip route get 10.4.0.1
> > 10.4.0.1 via 10.4.0.1 dev test table local src 10.4.0.2 
> >     cache 
> > # !?? why isn't table 50 used as, I believe, it should. And why  
> 
> The default rule set has the local table at priority 0 so all lookups
> hit that table first:
> 
> # ip ru ls
> 0:	from all lookup local
> 32766:	from all lookup main
> 32767:	from all lookup default
> 
> For some reason it hits a match doing the lookup in the merged
> local/main fib_table for this ip route get.
> 
> > does adding a rule "fixes" it :
> > 
> > sh-4.4# ip rule add prio 55555  
> 
> Adding this rule causes the local and main tables to be split into
> actual separate fib tables. Doing so causes the lookup in the local
> table to fail, so the lookup continues to the next rule -- which is
> your prio 50 table 50 rule.
> 
> I did not look into why the earlier rule addition did not cause the
> unmerge to happen -- it should have.

Thanks, (I feel like) I understand what's happening now.


> > sh-4.4# ip route get 10.4.0.1
> > 10.4.0.1 dev test table 50 src 10.4.0.2 
> >     cache 
> > # deleting the new rule makes no difference. It could even have been
> > done right after adding it. It's like it just triggered something
> > (reload, cache flushed, ...)
> > As seen I did flush cached routes as mentionned in the man page, I
> > don't know of anything else that would need done to "refresh"
> > things?
> > 
> > Any ideas as to why this is happening? Should this work as I expect
> > it, or is there anything I'm doing wrong?  
> 
> For your purposes now this should fix the problem for you:
> 
> ip ru del from all lookup local
> ip ru add prio 32765 from all lookup local

Indeed, if I first delete the rule for lookup local and recreate it
w/ higher prio than my "lookup 50", then no more issue.

Thanks a lot!

Re: Bug w/ (policy) routing

[David Ahern]

On 1/1/17 12:52 PM, Olivier Brunel wrote:
> Indeed, if I first delete the rule for lookup local and recreate it
> w/ higher prio than my "lookup 50", then no more issue.

After the unshare or when creating a new network namespace, bringing the lo device up will create the local table and the rest of the commands will work properly. ie., instead of moving the local rule you can run:

unshare -n bash

ip li set lo up
ip rule add table 50 prio 50
ip link add test type veth peer name test2
...

-----

Alex: 

The order of commands is influencing whether the unmerge succeeds or not which is wrong. I took a quick look and I don't see a simple solution to this. Effectively:

Adding a rule before bringing up any interface does not unmerge the tables:
$ unshare -n bash
$ ip rule add table 50 prio 50
$ ip li set lo up

In fib_unmerge(), fib_new_table(net, RT_TABLE_LOCAL) returns null.


Where the reverse order works:
$ unshare -n bash
$ ip li set lo up
$ ip rule add table 50 prio 50

David

Re: Bug w/ (policy) routing

[Olivier Brunel]

On Mon, 2 Jan 2017 09:48:12 -0700
David Ahern <dsa@cumulusnetworks.com> wrote:

> On 1/1/17 12:52 PM, Olivier Brunel wrote:
> > Indeed, if I first delete the rule for lookup local and recreate it
> > w/ higher prio than my "lookup 50", then no more issue.  
> 
> After the unshare or when creating a new network namespace, bringing
> the lo device up will create the local table and the rest of the
> commands will work properly. ie., instead of moving the local rule
> you can run:

Indeed, and that's a much better solution for me, since I bring lo up
anyways, I might as well do it first. Thank you.


> unshare -n bash
> 
> ip li set lo up
> ip rule add table 50 prio 50
> ip link add test type veth peer name test2
> ...
> 
> -----
> 
> Alex: 
> 
> The order of commands is influencing whether the unmerge succeeds or
> not which is wrong. I took a quick look and I don't see a simple
> solution to this. Effectively:
> 
> Adding a rule before bringing up any interface does not unmerge the
> tables: $ unshare -n bash
> $ ip rule add table 50 prio 50
> $ ip li set lo up
> 
> In fib_unmerge(), fib_new_table(net, RT_TABLE_LOCAL) returns null.
> 
> 
> Where the reverse order works:
> $ unshare -n bash
> $ ip li set lo up
> $ ip rule add table 50 prio 50
> 
> David