Re: ip rule iif oif and vrf

[Stephen Suryaputra <ssuryaextr@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3821 bytes --]

On Thu, Sep 24, 2020 at 08:41:54AM -0600, David Ahern wrote:
> > We have multiple options on the table right now. One that can be done
> > without writing any code is to use an nft prerouting rule to mark
> > the packet with iif equals the tunnel and use ip rule fwmark to lookup
> > the right table.
> > 
> > ip netns exec r0 nft add table ip c2t
> > ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> > ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
> > ip netns exec r0 ip rule add fwmark 101 table 10 pref 999
> > 
> > ip netns exec r1 nft add table ip c2t
> > ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
> > ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
> > ip netns exec r1 ip rule add fwmark 101 table 10 pref 999
> > 
> > But this doesn't seem to work on my Ubuntu VM with the namespaces
> > script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
> > does work on our target, based on 4.14 kernel.
> 
> add debugs to net/core/fib_rules.c, fib_rule_match() to see if
> flowi_mark is getting set properly. There could easily be places that
> are missed. Or if it works on one setup, but not another compare sysctl
> settings for net.core and net.ipv4

Sorry, I got side-tracked. Coming back to this: I made a mistake in the
ip rule fwmark pref in the script. I have fixed it and the script is
attached (gre_setup_nft.sh). It has the nft and ip rule commands above.
The ping between h0 and h1 works.

> > We also notice though in on the target platform that the ip rule fwmark
> > doesn't seem to change the skb->dev to the vrf of the lookup table.
> 
> not following that statement. fwmark should be marking the skb, not
> changing the skb->dev.

Yes and it causes the ping between h0 and r1 r1_v11 to not work, e.g.:

ip netns exec h0 ping -c 1 11.0.0.1

Similarly, ping between r0_v00 and r1_v11 also does not work:

ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1

> > E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
> > the reply is generated but the originating ping application doesn't get
> > the packet.  I suspect it is caused by the socket is bound to the tenant
> > vrf. I haven't been able to repro this because of the problem with the
> > nft approach above.

To illustrate my statements above, this is what I did:
ip netns exec r1 sysctl -w net.ipv4.fwmark_reflect=1
ip netns exec h0 ping -c 1 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=63 time=0.079 ms

The ping between h0 and r1 r1_v11 works, but it still doesn't work for this:
ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1

eventhough the reply is received by gre01:
ip netns exec r0 tcpdump -nexi gre01
22:10:57.173680 Out ethertype IPv4 (0x0800), length 100: 10.0.0.1 > 11.0.0.1: ICMP echo request, id 3803, seq 1, length 64
	0x0000:  4500 0054 1d2a 4000 4001 087e 0a00 0001
	0x0010:  0b00 0001 0800 a410 0edb 0001 b13a 755f
	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
	0x0050:  3435 3637
22:10:57.173724  In ethertype IPv4 (0x0800), length 100: 11.0.0.1 > 10.0.0.1: ICMP echo reply, id 3803, seq 1, length 64
	0x0000:  4500 0054 c709 0000 4001 9e9e 0b00 0001
	0x0010:  0a00 0001 0000 ac10 0edb 0001 b13a 755f
	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
	0x0050:  3435 3637

In summary the question is: should ip rule with action FR_ACT_TO_TBL
also change the skb->dev to the right l3mdev?

Thank you,

Stephen.

[-- Attachment #2: gre_setup_nft.sh --]
[-- Type: application/x-sh, Size: 5707 bytes --]