ixgbe: SR-IOV, macvlan filter on VFs

ixgbe: SR-IOV, macvlan filter on VFs

[Yoann Juet]

Hi all,

We are trying to make VRRP with VMAC address work on VMs using SR-IOV. 
The well known Keepalived VRRP framework implements such a feature with 
a macvlan device per physical interface. In our setup, this means we get 
a new macvlan device per VF. To make it partially work, we first have to 
disable the ixgbe anti-spoofing feature on the PFs that are involved:

ip link set dev <ethX> vf <x> spoofchk off

Now, VIPs' Virtual Mac Address are known from clients (arp answers are 
transmitted). But this is not enough ; VIPs are still not reachable from 
clients as they are attached to a macvlan device. Each time a VMAC is 
set on a macvlan device, we get the following messages:

[674943.437989] ixgbe 0000:04:00.0 eth2: VF 0 requested MACVLAN filter 
but is administratively denied
[674943.458875] ixgbe 0000:04:00.0 eth2: VF 1 requested MACVLAN filter 
but is administratively denied

Looks like macvlan on VF only works for outbound traffic from VMs. 
Inbound traffic is filtered. Is there a solution to disable macvlan 
filtering on a VF basis ?

Thanks,
Best Regards,

Re: ixgbe: SR-IOV, macvlan filter on VFs

[John Fastabend]

On 08/14/2014 05:04 AM, Yoann Juet wrote:
> Hi all,
>
> We are trying to make VRRP with VMAC address work on VMs using SR-IOV.
> The well known Keepalived VRRP framework implements such a feature with
> a macvlan device per physical interface. In our setup, this means we get
> a new macvlan device per VF. To make it partially work, we first have to
> disable the ixgbe anti-spoofing feature on the PFs that are involved:
>
> ip link set dev <ethX> vf <x> spoofchk off
>
> Now, VIPs' Virtual Mac Address are known from clients (arp answers are
> transmitted). But this is not enough ; VIPs are still not reachable from
> clients as they are attached to a macvlan device. Each time a VMAC is
> set on a macvlan device, we get the following messages:
>
> [674943.437989] ixgbe 0000:04:00.0 eth2: VF 0 requested MACVLAN filter
> but is administratively denied
> [674943.458875] ixgbe 0000:04:00.0 eth2: VF 1 requested MACVLAN filter
> but is administratively denied
>
> Looks like macvlan on VF only works for outbound traffic from VMs.
> Inbound traffic is filtered. Is there a solution to disable macvlan
> filtering on a VF basis ?
>
> Thanks,
> Best Regards,
>
> --

hmm this should work I think.

Did you set the VF mac address at some point with,

	ip link set dev DEVICE vf NUM mac ADDR

If not this how did you setup the virtual functions? Manually via
sriov_numvfs? Or via libvirt or other library. Basically there is a
check in the driver to see if the MAC was set via the physical function
and in this case the driver doesn't allow the VF to add/remove any
MAC addresses on the rx filters.

In the ixgbe driver if ixgbe_ndo_set_vf_mac is ever called it sets
the pf_set_mac bit (this is the check) and denies any further mac
updates. This function gets called when ever a netlink IFLA_VF_MAC
attribute is setup. This is what is invoked by the above 'ip link'
command.


Thanks,
John

-- 
John Fastabend         Intel Corporation

Re: ixgbe: SR-IOV, macvlan filter on VFs

[Yoann Juet]

> hmm this should work I think.
>
> Did you set the VF mac address at some point with,
>
>      ip link set dev DEVICE vf NUM mac ADDR
>
> If not this how did you setup the virtual functions? Manually via
> sriov_numvfs? Or via libvirt or other library.

I'm using libvirt with interface type 'network' or 'hostdev'. The mac 
address is optional within this configuration block. But, If omitted, 
it's automatically generated. Even if I could set the mac address via 
the 'ip link' command (not very practical with libvirt but possible), 
Keepalived still needs two mac addresses per VF. From the guest perspective:
	- ethX: @mac1, @ip1
	- macvlanX<->ethX: @mac2 (vmac), @ip2 (vip)

Correct me if I'm wrong, the command 'ip link set dev DEVICE vf NUM mac 
ADDR' can only set one mac per VF.

Thanks,
Regards,

Re: ixgbe: SR-IOV, macvlan filter on VFs

[John Fastabend]

On 08/14/2014 08:27 AM, Yoann Juet wrote:
>> hmm this should work I think.
>>
>> Did you set the VF mac address at some point with,
>>
>>      ip link set dev DEVICE vf NUM mac ADDR
>>
>> If not this how did you setup the virtual functions? Manually via
>> sriov_numvfs? Or via libvirt or other library.
> 
> I'm using libvirt with interface type 'network' or 'hostdev'. The mac
> address is optional within this configuration block. But, If omitted,
> it's automatically generated. Even if I could set the mac address via
> the 'ip link' command (not very practical with libvirt but possible),
> Keepalived still needs two mac addresses per VF. From the guest
> perspective:
>     - ethX: @mac1, @ip1
>     - macvlanX<->ethX: @mac2 (vmac), @ip2 (vip)
> 
> Correct me if I'm wrong, the command 'ip link set dev DEVICE vf NUM mac ADDR' can only set one mac per VF.
> 

Right this seems to be the case. My guess is libvirt uses the IFLA_VF_MAC
attribute to set the MAC from the host side. After this is done the PF
driver will deny any additional MAC requests from the VF which is where
you get that error message.

I need to check the libvirt theory but if that is the case I'm not sure
what the best fix off hand is. Maybe an option to libvirt not to set the
MAC via the PF. The other option would be to add another knob to allow the
setting of MAC addrs from the VF via 'ip link' similar to the TX spoofing
bit already in place. Both would presumably resolve your use case.

.John

Re: ixgbe: SR-IOV, macvlan filter on VFs

[Yoann Juet]

> Right this seems to be the case. My guess is libvirt uses the IFLA_VF_MAC
> attribute to set the MAC from the host side. After this is done the PF
> driver will deny any additional MAC requests from the VF which is where
> you get that error message.
>
> I need to check the libvirt theory but if that is the case I'm not sure
> what the best fix off hand is. Maybe an option to libvirt not to set the
> MAC via the PF. The other option would be to add another knob to allow the
> setting of MAC addrs from the VF via 'ip link' similar to the TX spoofing
> bit already in place. Both would presumably resolve your use case.
>
> .John

Thanks a lot for your assistance John. Hope that a fix will be found 
regarding this issue.

Yoann.