Re: vrf and multicast is broken in some cases

[Ido Schimmel <idosch@idosch.org>]

On Wed, Jan 15, 2020 at 12:00:08PM -0800, Ben Greear wrote:
> 
> 
> On 01/15/2020 11:19 AM, Ido Schimmel wrote:
> > On Wed, Jan 15, 2020 at 11:02:26AM -0800, Ben Greear wrote:
> > > 
> > > 
> > > On 01/15/2020 10:45 AM, David Ahern wrote:
> > > > On 1/15/20 10:57 AM, Ben Greear wrote:
> > > > > Hello,
> > > > > 
> > > > > We put two different ports into their own VRF, and then tried to run a
> > > > > multicast
> > > > > sender on one and receiver on the other.  The receiver does not receive
> > > > > anything.
> > > > > 
> > > > > Is this a known problem?
> > > > > 
> > > > > If we do a similar setup with policy based routing rules instead of VRF,
> > > > > then the multicast
> > > > > test works.
> > > > > 
> > > > 
> > > > It works for OSPF for example. I have lost track of FRR features that
> > > > use it, so you will need to specify more details.
> > > > 
> > > > Are the sender / receiver on the same host?
> > > 
> > > Yes, like eth2 sending to eth3, eth2 is associated with _vrf2, eth3 with _vrf3.
> > 
> > Two questions:
> > 
> > 1. Did you re-order the FIB rules so that l3mdev rule is before the main
> > table?
> 
> That seems OK:
> 
> [root@lf0313-6477 lanforge]# ip ru show
> 1000:	from all lookup [l3mdev-table]
> 1512:	from all lookup local
> 32766:	from all lookup main
> 32767:	from all lookup default
> 
> 
> > 2. Did you configure a default unreachable route in the VRF?
> 
> I did not have this, so maybe that is the issue.  This is my mcast
> transmitter table.
> 
> [root@lf0313-6477 lanforge]# ip route show table 10
> broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
> local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
> broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 
> When sniffing, I see IGMP group add/delete messages sent from the receiver
> towards the sender, but transmitted mcast frames are not seen on the rddVR0
> (veth mcast sender port).
> 
> > 
> > IIRC, locally generated multicast packets are forwarded according to the
> > unicast FIB rules, so if you don't have the unreachable route, it is
> > possible the packet is forwarded according to the default route in the
> > main table.
> 
> And now that is interesting.  When I sniff on eth0, which holds the default
> route outside of the VRFs, then I do see the mcast frames sent there.
> 
> I tried adding default routes, and now sure enough it starts working!

I'm not sure this is the correct way (David?). Can you try to delete
this default route and instead add a default unreachable route with an
high metric according to step 3 in Documentation/networking/vrf.txt:

"
3. Set the default route for the table (and hence default route for the VRF).
       ip route add table 10 unreachable default metric 4278198272

   This high metric value ensures that the default unreachable route can
   be overridden by a routing protocol suite.  FRRouting interprets
   kernel metrics as a combined admin distance (upper byte) and priority
   (lower 3 bytes).  Thus the above metric translates to [255/8192].
"

If I'm reading ip_route_output_key_hash_rcu() correctly, then the error
returned from fib_lookup() because of the unreachable route should allow
you to route the packet via the requested interface.

> 
> [root@lf0313-6477 lanforge]# ip route show table 10
> default via 7.7.1.1 dev rddVR0
> broadcast 7.7.1.0 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 7.7.1.0/24 dev rddVR0  scope link  src 7.7.1.2
> local 7.7.1.2 dev rddVR0  proto kernel  scope host  src 7.7.1.2
> broadcast 7.7.1.255 dev rddVR0  proto kernel  scope link  src 7.7.1.2
> 
> 
> I'll work on adding an un-reachable route when a real gateway is not
> configured...
> 
> Thanks for the hint, saved me lots of work!
> 
> --Ben
> 
> > 
> > > 
> > > I'll go poking at the code.
> > > 
> > > Thanks,
> > > Ben
> > > 
> > > --
> > > Ben Greear <greearb@candelatech.com>
> > > Candela Technologies Inc  http://www.candelatech.com
> > 
> 
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com