Re: VRF Issue Since kernel 5

[David Ahern <dsahern@gmail.com>]

On 3/10/20 2:47 PM, Maximilian Bosch wrote:
> Hi!
> 
> I suspect I hit the same issue which is why I decided to respond to this
> thread (if that's wrong please let me know).
> 
>> sudo sysctl -a | grep l3mdev
>>
>> If not,
>> sudo sysctl net.ipv4.raw_l3mdev_accept=1
>> sudo sysctl net.ipv4.udp_l3mdev_accept=1
>> sudo sysctl net.ipv4.tcp_l3mdev_accept=1
> 
> On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but
> I experience the same issue.
> 
>> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)
> 
> I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect
> that only TCP traffic hangs in the VRF. When I try to `ssh` through the
> VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine.
> 
> However, TCP traffic through a VRF works fine as well on 4.x (just tested this on
> 4.19.108 and 4.14.172).

functional test script under tools/testing/selftests/net covers VRF
tests and it ran clean for 5.4 last time I checked. There were a few
changes that went into 4.20 or 5.0 that might be tripping up this use
case, but I need a lot more information.

> 
> I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0).
> My main routing table has a default route via my WireGuard Gateway and I
> only route my WireGuard uplink through the VRF. With this approach I can
> make sure that all of my traffic goes through the VPN and only the
> UDP packets of WireGuard will be routed through the uplink network.

are you saying wireguard worked with VRF in the past but is not now?


> 
> As mentioned above, the WireGuard traffic works perfectly fine, but I
> can't access `<vpn-uplink>` via SSH:
> 
> ```
> $ ssh root@<vpn-uplink> -vvvv
> OpenSSH_8.2p1, OpenSSL 1.1.1d  10 Sep 2019
> debug1: Reading configuration data /home/ma27/.ssh/config
> debug1: /home/ma27/.ssh/config line 5: Applying options for *
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug1: /etc/ssh/ssh_config line 5: Applying options for *
> debug2: resolve_canonicalize: hostname <vpn-uplink> is address
> debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist
> debug2: ssh_connect_direct
> debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22.
> # Hangs here for a while
> ```
> 
> I get the following output when debugging this with `tcpdump`:
> 
> ```
> $ tcpdump -ni uplink tcp
> 20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0
> 20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0
> 20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0

that suggests not finding a matching socket, so sending a reset.

> 20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0
> 20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0
> 20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0
> ```
> 
> AFAICS every SYN will be terminated with an RST which is the reason why
> the connection hangs.
> 
> I can work around the issue by using `ip vrf exec`. However I get the
> following error (unless I run `ulimit -l 2048`):
> 
> ```
> Failed to load BPF prog: 'Operation not permitted'
> ```

'ip vrf exec' loads a bpf program and that requires locked memory, so
yes, you need to increase it.

Let's start with lookups:

perf record -e fib:* -a -g
<run test that fails, ctrl-c>
perf script

That shows the lookups (inputs, table id, result) and context (stack
trace). That might give some context.