sr-iov and bridges (mlx4)

sr-iov and bridges (mlx4)

[Matthew Monaco]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

Hello. I have a Mellanox ConnectX-3 Pro EN (MCX314A-BCCT). I'm only using a
single port so it must provide IP for my host as well as connectivity for VMs.
SR-IOV VFs are working great, my KVM VMs have Ethernet and RDMA.

However, I also want to support virtio VMs. Assuming eth0 is the first port on
my mlx nic, I've tried placing VMs on a bridge with the primary physical
interface, and giving an IP for management to a VF:

br0
|--- eth0
|--- VM
|--- VM
vf0 (IP)
vf1 -> VM
vf2 -> VM
vf3 -> VM

I've tried placing VMs on a bridge with one of the VFs and using the primary
iface for IP.

eth0 (IP)
br0
|--- vf0
|--- VM
|--- VM
vf1 -> VM
vf2 -> VM
vf3 -> VM

And I've also tried using a veth pair to really spread things out:

br0 (IP)
|--- eth0
|--- veth-a
br1       |
|--- veth-b
|--- VM
|--- VM
vf1 -> VM
vf2 -> VM
vf3 -> VM

In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound
DHCP from the virtio VMs work fine, but inbound frames are not making it back to
the VM.

Is there a know limitation of mixing SR-IOV and bridges in general? Does the
SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?

Thanks!
Matt


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: sr-iov and bridges (mlx4)

[Or Gerlitz]

On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco
<Matthew.Monaco@colorado.edu> wrote:
[...]
> In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound
> DHCP from the virtio VMs work fine, but inbound frames are not making it back to
> the VM.
[...]
> Is there a know limitation of mixing SR-IOV and bridges in general? Does the
> SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?

It would be a bit hard for bridge based promiscuous environment to work OOB
for one of the functions (PF or VF) in SRIOV, this is generic issue,
and not related
to specific vendor.

You need to use the bridge (8) tool (part of iproute2) and add the P.V
VM MACs to
the PF interface as "self" see some slides  (21/22/23) from netdev 0.1
that deal with that
https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf

Or.

It also possible to mark one bridge port (the PF) as non-promiscuous,
but I haven't
played with that myself yet (slide 32), so can't just send you doing it...

Re: sr-iov and bridges (mlx4)

[Matthew Monaco]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

On 09/20/2015 05:01 AM, Or Gerlitz wrote:
> On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco
> <Matthew.Monaco@colorado.edu> wrote:
> [...]
>> In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound
>> DHCP from the virtio VMs work fine, but inbound frames are not making it back to
>> the VM.
> [...]
>> Is there a know limitation of mixing SR-IOV and bridges in general? Does the
>> SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...?
> 
> It would be a bit hard for bridge based promiscuous environment to work OOB
> for one of the functions (PF or VF) in SRIOV, this is generic issue,
> and not related
> to specific vendor.
> 
> You need to use the bridge (8) tool (part of iproute2) and add the P.V
> VM MACs to
> the PF interface as "self" see some slides  (21/22/23) from netdev 0.1
> that deal with that
> https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf
> 
> Or.
> 
> It also possible to mark one bridge port (the PF) as non-promiscuous,
> but I haven't
> played with that myself yet (slide 32), so can't just send you doing it...
>

Many thanks, that helps. Now to figure out the best approach for shoving this
into OpenStack =)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: sr-iov and bridges (mlx4)

[Nieścierowicz Adam]

Hi Matthew,
in near future i want to do exactly the same, if you make progress with 
SR-IOV+Bridge+OpenStack please leave here some advice.

---
Pozdrawiam
Adam Nieścierowicz

W dniu 20.09.2015 18:21, Matthew Monaco napisał(a):

> On 09/20/2015 05:01 AM, Or Gerlitz wrote: On Sun, Sep 20, 2015 at 2:58 
> AM, Matthew Monaco
> <Matthew.Monaco@colorado.edu> wrote:
> [...] In all cases, VMs with SR-IOV work fine, IP on the host works 
> fine, outbound
> DHCP from the virtio VMs work fine, but inbound frames are not making 
> it back to
> the VM. [...] Is there a know limitation of mixing SR-IOV and bridges 
> in general? Does the
> SR-IOV switch specific to the mlx4 hardware not work well with linux 
> bridges? ...?
> It would be a bit hard for bridge based promiscuous environment to work 
> OOB
> for one of the functions (PF or VF) in SRIOV, this is generic issue,
> and not related
> to specific vendor.
> 
> You need to use the bridge (8) tool (part of iproute2) and add the P.V
> VM MACs to
> the PF interface as "self" see some slides (21/22/23) from netdev 0.1
> that deal with that
> https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf [1]
> 
> Or.
> 
> It also possible to mark one bridge port (the PF) as non-promiscuous,
> but I haven't
> played with that myself yet (slide 32), so can't just send you doing 
> it...

  Many thanks, that helps. Now to figure out the best approach for 
shoving this
  into OpenStack =)



Links:
------
[1] https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf

Re: sr-iov and bridges (mlx4)

[Matthew Monaco]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On 09/20/2015 12:18 PM, Nieścierowicz Adam wrote:
> Hi Matthew,
> in near future i want to do exactly the same, if you make progress with
> SR-IOV+Bridge+OpenStack please leave here some advice.
> 

I wrote a shell script which polls (~15s) each bridge /sys/class/net/brq*, and
for each bridge figures out the uplink (not named
/sys/class/net/brq*/brif/tap*). Then, for each port
/sys/class/net/<bridge>/brif/tap*, determine the VM mac address and add an fdb
entry if it doesn't already exist. In my environment, it seems the VM mac is the
tap mac s/^fe:/fa:/.

This is a little messier than the non-promsicuous bridge option, but I wasn't
able to get that working on CentOS7/kernel-ml-4.2/iproute-3.10. But either way,
this won't work well if you're trying to do nested virt, which thankfully I
don't need at this time.

If you're interested I can attach the script and systemd unit. Otherwise, I'll
be looking to try to get this handled properly by neutron-linuxbridge-agent.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]