[PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[Antoine Tenart]

VRF devices are prevented from being added to upper devices since commit
1017e0987117 ("vrf: prevent adding upper devices") as they set the
IFF_NO_RX_HANDLER flag. However attaching a VRF to an OVS bridge is a
valid use case[1].

Allow a VRF device to be attached to an OVS bridge by having an OVS
specific tweak. This approach allows not to change a valid logic
elsewhere and the IFF_NO_RX_HANDLER limitation still applies for non-OVS
upper devices, even after a VRF was unlinked from an OVS bridge.

(Patch not sent as a fix as the commit introducing the limitation is not
recent).

[1] https://ltomasbo.wordpress.com/2021/06/25/openstack-networking-with-evpn/

Signed-off-by: Antoine Tenart <atenart@kernel.org>
---

Hi all,

I thought about other ways to fix this but did not want to add yet
another flag, nor to add specific logic outside of net/openvswitch/. A
custom netdev_rx_handler_register having priv_flags as a parameter could
also have been added, but again that seemed a bit invasive.

There might be questions about the setup in which a VRF is linked to an
OVS bridge; I cc'ed Luis Tomás who wrote the article.

Thanks,
Antoine

 net/openvswitch/vport-netdev.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/net/openvswitch/vport-netdev.c b/net/openvswitch/vport-netdev.c
index 8e1a88f13622..e76b2477d384 100644
--- a/net/openvswitch/vport-netdev.c
+++ b/net/openvswitch/vport-netdev.c
@@ -75,6 +75,7 @@ static struct net_device *get_dpdev(const struct datapath *dp)
 
 struct vport *ovs_netdev_link(struct vport *vport, const char *name)
 {
+	unsigned int saved_flags;
 	int err;
 
 	vport->dev = dev_get_by_name(ovs_dp_get_net(vport->dp), name);
@@ -98,8 +99,17 @@ struct vport *ovs_netdev_link(struct vport *vport, const char *name)
 	if (err)
 		goto error_unlock;
 
+	/* While IFF_NO_RX_HANDLER is rightly set for l3 masters (VRF) as they
+	 * don't work with upper devices, they can be attached to OVS bridges.
+	 */
+	saved_flags = vport->dev->priv_flags;
+	if (netif_is_l3_master(vport->dev))
+		vport->dev->priv_flags &= ~IFF_NO_RX_HANDLER;
+
 	err = netdev_rx_handler_register(vport->dev, netdev_frame_hook,
 					 vport);
+	vport->dev->priv_flags = saved_flags;
+
 	if (err)
 		goto error_master_upper_dev_unlink;
 
-- 
2.31.1

Re: [PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[David Ahern]

On 9/20/21 9:34 AM, Antoine Tenart wrote:
> There might be questions about the setup in which a VRF is linked to an
> OVS bridge; I cc'ed Luis Tomás who wrote the article.

My head just exploded. You want to make an L3 device a port of an L2
device.

Can someone explain how this is supposed to work and why it is even
needed? ie., given how an OVS bridge handles packets and the point of a
VRF device (to direct lookups to a table), the 2 are at odds in my head.

Re: [PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[Luis Tomas Bolivar]

On Mon, Sep 20, 2021 at 5:45 PM David Ahern <dsahern@gmail.com> wrote:
>
> On 9/20/21 9:34 AM, Antoine Tenart wrote:
> > There might be questions about the setup in which a VRF is linked to an
> > OVS bridge; I cc'ed Luis Tomás who wrote the article.
>
> My head just exploded. You want to make an L3 device a port of an L2
> device.
>
> Can someone explain how this is supposed to work and why it is even
> needed? ie., given how an OVS bridge handles packets and the point of a
> VRF device (to direct lookups to a table), the 2 are at odds in my head.
>

Hi David,

Thanks for your comment. And yes you are right, this probably is a bit
of an odd setup. That said, OVS is not pure L2 as it knows about IPs
and it is doing virtual routing too (we can say it is 2.5 xD)

What we want to achieve is something similar to what is shown in slide 100
here http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf, but instead
of connecting the VRF bridge directly to containers, we have a single ovs
bridge (where the OpenStack VMs are connected to) where we connect the
vrfs in different (ovs) ports (so that the traffic in the way out of OVS can be
redirected to the right VRF).

The initial part is pretty much the same as in the slide 100:
1) creating the vrf
   - ip link add vrf-1001 type vrf table 1001
2) vxlan device, in our case associated to the loopback,
for ECMP (instead of associate both nics/vlan devices to the VRF)
   - ip link add vxlan-1001 type vxlan id 1001 dstport 4789 local L_IP
nolearning
3) create the linux bridge device
   - ip link add name br-1001 type bridge stp_state 0
4) link the 3 above
   - ip link set br-1001 master vrf-1001 (bridge to vrf)
   - ip link set vxlan-1001 master br-1001 (vxlan to bridge)

Then, I'm attaching the vrf device also as an ovs bridge port, so that
traffic (together with some ip routes in the vrf routing table) can be
redirected
to OVS, and the (OpenStack) virtual networking happens there
(br-ex is the ovs bridge)
   - ovs-vsctl add-port br-ex vrf-1001
   - ip route show vrf vrf-1001
       10.0.0.0/26 via 172.24.4.146 dev br-ex
       (redirect traffic to OpenStack subnet 10.0.0.0/26 to br-ex)
       172.24.4.146 dev br-ex scope link

Perhaps there is a better way of connecting this?
I tried (without success) to create a veth device and set one end on
the linux bridge and the other on the OVS bridge.

Best regards,
Luis

Re: [PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[Luis Tomas Bolivar]

On Tue, Sep 21, 2021 at 10:12 AM Luis Tomas Bolivar <ltomasbo@redhat.com> wrote:
>
> On Mon, Sep 20, 2021 at 5:45 PM David Ahern <dsahern@gmail.com> wrote:
> >
> > On 9/20/21 9:34 AM, Antoine Tenart wrote:
> > > There might be questions about the setup in which a VRF is linked to an
> > > OVS bridge; I cc'ed Luis Tomás who wrote the article.
> >
> > My head just exploded. You want to make an L3 device a port of an L2
> > device.
> >
> > Can someone explain how this is supposed to work and why it is even
> > needed? ie., given how an OVS bridge handles packets and the point of a
> > VRF device (to direct lookups to a table), the 2 are at odds in my head.
> >
>
> Hi David,
>
> Thanks for your comment. And yes you are right, this probably is a bit
> of an odd setup. That said, OVS is not pure L2 as it knows about IPs
> and it is doing virtual routing too (we can say it is 2.5 xD)
>
> What we want to achieve is something similar to what is shown in slide 100
> here http://schd.ws/hosted_files/ossna2017/fe/vrf-tutorial-oss.pdf, but instead
> of connecting the VRF bridge directly to containers, we have a single ovs
> bridge (where the OpenStack VMs are connected to) where we connect the
> vrfs in different (ovs) ports (so that the traffic in the way out of OVS can be
> redirected to the right VRF).
>
> The initial part is pretty much the same as in the slide 100:
> 1) creating the vrf
>    - ip link add vrf-1001 type vrf table 1001
> 2) vxlan device, in our case associated to the loopback,
> for ECMP (instead of associate both nics/vlan devices to the VRF)
>    - ip link add vxlan-1001 type vxlan id 1001 dstport 4789 local L_IP
> nolearning
> 3) create the linux bridge device
>    - ip link add name br-1001 type bridge stp_state 0
> 4) link the 3 above
>    - ip link set br-1001 master vrf-1001 (bridge to vrf)
>    - ip link set vxlan-1001 master br-1001 (vxlan to bridge)
>
> Then, I'm attaching the vrf device also as an ovs bridge port, so that
> traffic (together with some ip routes in the vrf routing table) can be
> redirected
> to OVS, and the (OpenStack) virtual networking happens there
> (br-ex is the ovs bridge)
>    - ovs-vsctl add-port br-ex vrf-1001
>    - ip route show vrf vrf-1001
>        10.0.0.0/26 via 172.24.4.146 dev br-ex
>        (redirect traffic to OpenStack subnet 10.0.0.0/26 to br-ex)
>        172.24.4.146 dev br-ex scope link
>
> Perhaps there is a better way of connecting this?
> I tried (without success) to create a veth device and set one end on
> the linux bridge and the other on the OVS bridge.

Follow up on this. I found the mistake I was making on the veth-pair
addition configuration (ovs flow was setting the wrong mac address
before sending the traffic through the veth device to the vrf). And it
indeed works connecting the VRF to the OVS bridge by using a veth pair
instead of directly plugin the VRF device as an OVS port.

>
> Best regards,
> Luis



-- 
LUIS TOMÁS BOLÍVAR
Principal Software Engineer
Red Hat
Madrid, Spain
ltomasbo@redhat.com

Re: [PATCH net-next] openvswitch: allow linking a VRF to an OVS bridge

[Antoine Tenart]

Hello,

Quoting Luis Tomas Bolivar (2021-09-21 13:20:08)
> 
> Follow up on this. I found the mistake I was making on the veth-pair
> addition configuration (ovs flow was setting the wrong mac address
> before sending the traffic through the veth device to the vrf). And it
> indeed works connecting the VRF to the OVS bridge by using a veth pair
> instead of directly plugin the VRF device as an OVS port.

Great! This means there is no need for this patch I believe, OVS bridge
is not different in the end.

Thanks,
Antoine