Re: VRF with enslaved L3 enabled bridge

[David Ahern <dsa@cumulusnetworks.com>]

On 7/19/18 8:19 PM, D'Souza, Nelson wrote:
> Hi,
> 
>  
> 
> I'm seeing the following issue on a system running a 4.14.52 Linux kernel.
> 
>  
> 
> With an eth interface enslaved to a VRF device, pings sent out on the
> VRF to an neighboring host are successful. But, with an eth interface
> enslaved to a L3 enabled bridge (mgmtbr0), and the bridge enslaved to a
> l3mdev VRF (mgmtvrf), the pings sent out on the VRF are not received
> back at the application level.

you mean this setup:
    eth1 (ingress port) -> br0 (bridge) -> red (vrf)

IP address on br0:

9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master
red state UP group default qlen 1000
    link/ether 02:e0:f9:1c:00:37 brd ff:ff:ff:ff:ff:ff
    inet 10.100.1.4/24 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0:f9ff:fe1c:37/64 scope link
       valid_lft forever preferred_lft forever

And then ping a neighbor:

# ping -I red -c1 -w1 10.100.1.254
ping: Warning: source address might be selected on device other than red.
PING 10.100.1.254 (10.100.1.254) from 10.100.1.4 red: 56(84) bytes of data.
64 bytes from 10.100.1.254: icmp_seq=1 ttl=64 time=0.810 ms

--- 10.100.1.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.810/0.810/0.810/0.000 ms

> 
>  
> 
> ICMP Echo requests are successfully sent out on the mgmtvrf device to a
> neighboring host. However, ICMP echo replies that are received back from
> the neighboring host via the eth and mgmtbr0 interfaces are not seen at
> the vrf device level and therefore fail to be delivered locally to the
> ping application.

Does tcpdump on each level show the response? tcpdump on eth, tcpdump on
bridge and tcpdump on the vrf device?

> 
>  
> 
> The following LOG rules were added to the raw table, prerouting chain
> and the filter table, OUTPUT chains:
> 
>  
> 
> root@x10sdv-4c-tln4f:~# iptables -t raw -S PREROUTING
> 
> -P PREROUTING ACCEPT
> 
> -A PREROUTING -s 10.32.8.135/32 -i mgmtbr0 -j LOG
> 
> -A PREROUTING -s 10.32.8.135/32 -i mgmtvrf -j LOG
> 
>  
> 
> root@x10sdv-4c-tln4f:~# iptables -S OUTPUT
> 
> -P OUTPUT ACCEPT
> 
> -A OUTPUT -o mgmtvrf -j LOG
> 
> -A OUTPUT -o mgmtbr0 -j LOG
> 
>  
> 
> Pings are sent on the management VRF to a neighboring host (10.32.8.135)
> and the netfilter logs included below:
> 
> Note, that in the logs, ICMP echo requests are sent out on the mgmtvrf
> and match the output rules for mgmvrf and mgmtbr0, but the ICMP echo
> replies are only seen on mgmtbr0, not on mgmtvrf
> 
>  
> 
> root@x10sdv-4c-tln4f:~# ping 10.32.8.135 -I mgmtvrf -c 1
> 
> PING 10.32.8.135  (10.32.8.135):
> 
> 56 data bytes
> 
> [ 2679.683027] IN= OUT=mgmtvrf SRC=10.33.96.131 DST=10.32.8.135 LEN=84
> TOS=0x00 PREC=0x00 TTL=64 ID=23921 DF PROTO=ICMP TYPE=8 CODE=0 ID=32610
> SEQ=0   <<< ICMP echo sent on mgmtvrf
> 
> [ 2679.697560] IN= OUT=mgmtbr0 SRC=10.33.96.131 DST=10.32.8.135 LEN=84
> TOS=0x00 PREC=0x00 TTL=64 ID=23921 DF PROTO=ICMP TYPE=8 CODE=0 ID=32610
> SEQ=0   <<< ICMP echo sent on mgmtbr0
> 
> [ 2679.713312] IN=mgmtbr0 OUT= PHYSIN=ethUSB
> MAC=c0:56:27:90:4f:75:c4:7d:4f:bb:02:e7:08:00 SRC=10.32.8.135
> DST=10.33.96.131 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=64949 PROTO=ICMP
> TYPE=0 CODE=0 ID=32610 SEQ=0     <<< ICMP echo reply rcvd on mgmtbr0,
> but not on mgmtvrf
> 
>  
> 
> --- 10.32.8.135 ping statistics ---
> 
> 1 packets transmitted, 0 packets received, 100% packet loss     <<<<
> ping failed
> 
>  
> 
> I’d like to know if this is an outstanding/resolved issue.
> 

This one works (see above), so I suspect it is something with your setup.