ixgbe: macvlan on PF/VF when SRIOV is enabled

ixgbe: macvlan on PF/VF when SRIOV is enabled

[Shirley Ma]

Hello Jeff,

macvlan doesn't work on PF when SRIOV is enabled. Creating macvlan has
been successful, but ping (icmp request) goes to VF interface not
PF/macvlan even arp entry is correct. I patched ixgbe driver, and
macvlan/PF has worked with the patch. But I am not sure whether it is
right since I don't have the HW spec. What I did for ixgbe driver was: 

1. PF's rar index is 0, VMDQ index is adatper->num_vfs;
2. VF's rar is based on rar_used_count and mc_addr_in_rar_count, VMDQ
index is ;
3. PF's secondary addresses is PF's rar index + i, VMDQ index is
adapter->num_vfs.


Before I submit the patch, I want to understand the right index
assignment for both rar index and VMDQ index, when SRIOV enabled:
1. VMDQ index for PF is adapter->num_vfs, or 0? rar index is 0?
2. PF's secondary address rar index is based on
rar_used_count/mc_addr_in_rar_count?
2. VF's VPDQ index is based on vf number?
3. VF's rar index is vf + 1, or should be based on rar_used_count?

I am also working on macvlan on VF. The question here is whether macvlan
on VF should work or not? Looks like ixgbevf secondary addresses are not
in receiver address filter, so macvlan on VF doesn't work.

Thanks
Shirley

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Rose, Gregory V]

>-----Original Message-----
>From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
>On Behalf Of Shirley Ma
>Sent: Friday, May 21, 2010 1:31 PM
>To: Kirsher, Jeffrey T
>Cc: davem@davemloft.net; kvm@vger.kernel.org; netdev@vger.kernel.org;
>e1000-devel@lists.sourceforge.net
>Subject: ixgbe: macvlan on PF/VF when SRIOV is enabled
>
>Hello Jeff,
>
>macvlan doesn't work on PF when SRIOV is enabled. Creating macvlan has
>been successful, but ping (icmp request) goes to VF interface not
>PF/macvlan even arp entry is correct. I patched ixgbe driver, and
>macvlan/PF has worked with the patch. But I am not sure whether it is
>right since I don't have the HW spec. What I did for ixgbe driver was:
>
>1. PF's rar index is 0, VMDQ index is adatper->num_vfs;
>2. VF's rar is based on rar_used_count and mc_addr_in_rar_count, VMDQ
>index is ;
>3. PF's secondary addresses is PF's rar index + i, VMDQ index is
>adapter->num_vfs.

As of 2.6.34 the ixgbe driver does not support multiple queues for macvlan.
Support for multiple queues for macvlan will come in a subsequent release.

>
>
>Before I submit the patch, I want to understand the right index
>assignment for both rar index and VMDQ index, when SRIOV enabled:
>1. VMDQ index for PF is adapter->num_vfs, or 0? rar index is 0?
>2. PF's secondary address rar index is based on
>rar_used_count/mc_addr_in_rar_count?
>2. VF's VPDQ index is based on vf number?
>3. VF's rar index is vf + 1, or should be based on rar_used_count?
>
>I am also working on macvlan on VF. The question here is whether macvlan
>on VF should work or not? Looks like ixgbevf secondary addresses are not
>in receiver address filter, so macvlan on VF doesn't work.

The VF driver does not support macvlan.  Future releases may but there
are no immediate plans to support it.

- Greg Rose
Intel Corp.
Lan Access Division

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Shirley Ma]

Hello Greg,

Thanks for your prompt response.

On Sat, 2010-05-22 at 10:53 -0700, Rose, Gregory V wrote:
> As of 2.6.34 the ixgbe driver does not support multiple queues for
> macvlan.
> Support for multiple queues for macvlan will come in a subsequent
> release.

When it might happen? I will double check my test to see whether macvlan
was multiple queue or not. Then submitting my experimental patch for
review.

> The VF driver does not support macvlan.  Future releases may but there
> are no immediate plans to support it.

When it might be support in future. For performance reason, we are
interested in macvlan + VF for multiples VMs.

One more question here: Does VF support promiscuous mode? I don't see
the flag in ixgbevf driver.

Thanks
Shirley

Re: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Rose, Gregory V]

>-----Original Message-----
>From: Shirley Ma [mailto:mashirle@us.ibm.com]
>Sent: Monday, May 24, 2010 10:09 AM
>To: Rose, Gregory V
>Cc: Kirsher, Jeffrey T; davem@davemloft.net; kvm@vger.kernel.org;
>netdev@vger.kernel.org; e1000-devel@lists.sourceforge.net
>Subject: RE: ixgbe: macvlan on PF/VF when SRIOV is enabled
>
>Hello Greg,
>
>Thanks for your prompt response.
>
>On Sat, 2010-05-22 at 10:53 -0700, Rose, Gregory V wrote:
>> As of 2.6.34 the ixgbe driver does not support multiple queues for
>> macvlan.
>> Support for multiple queues for macvlan will come in a subsequent
>> release.
>
>When it might happen?

Support for multiple queues in the PF driver is in the planning stage right now.  Hopefully we'll get it in sooner rather than later but I can't give any solid dates for it.

 I will double check my test to see whether macvlan
>was multiple queue or not.

Ah, so you just want to set multiple macvlan filters for the PF driver but aren't concerned about directing the traffic to different queues?  We haven't tested that in SR-IOV modes of operation but we can have a look at it.

Then submitting my experimental patch for
>review.

We look forward to it and will be happy to provide feedback.

>
>> The VF driver does not support macvlan.  Future releases may but there
>> are no immediate plans to support it.
>
>When it might be support in future. For performance reason, we are
>interested in macvlan + VF for multiples VMs.

There is a resource contention issue in this case.  There are 128 MAC filters available.  When VFs are allocated each will use a MAC filter entry.  In the case where the maximum number of VFs are allocated (63 in this case) there aren't a whole lot of MAC filters left over to spread across that many VFs.  Our view has been that it wouldn't be that much added value to support macvlan in the VFs but if there is a good case for it we'll consider it.

One thing you can do is allocate VFs and then load the VF driver in your host domain and then assign each of them a macvlan filter.  You'd get a similar effect.

>
>One more question here: Does VF support promiscuous mode? I don't see
>the flag in ixgbevf driver.

No, there is no per-VF support for promiscuous mode in the HW as such, however you could set the unicast hash table array to all ones and then set the VFs you want to be in promiscuous mode to accept untagged packets via the VM filtering and offload register at offset 0F000h.  However, re-provisioning the UTA for this purpose would preclude using it for the designed purpose.

I suggest that you pick up a copy of the developer's manual for the 82599 at the Intel developer's site.

http://developer.intel.com/products/ethernet/index.htm?iid=nc+ethernet#s1=all&s2=82576EB&s3=all

- Greg

------------------------------------------------------------------------------

_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Shirley Ma]

On Mon, 2010-05-24 at 10:54 -0700, Rose, Gregory V wrote:
> We look forward to it and will be happy to provide feedback.
I have submitted the patch to make macvlan on PF works when SRIOV is
enabled.

> One thing you can do is allocate VFs and then load the VF driver in
> your host domain and then assign each of them a macvlan filter.  You'd
> get a similar effect.

That's I am trying to make it work for macvlan on VFs in host domain. I
need to add VF secondary addresses in address filter, right?

Do you have any aggregation performance comparison between multiple
macvlans on PF and single macvlan per VF in host domain? I will run some
test to figure it out. If you have some data to share that would be
great.

Thanks
Shirley

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Rose, Gregory V]

>-----Original Message-----
>From: Shirley Ma [mailto:mashirle@us.ibm.com]
>Sent: Tuesday, May 25, 2010 2:37 AM
>To: Rose, Gregory V
>Cc: Kirsher, Jeffrey T; davem@davemloft.net; kvm@vger.kernel.org;
>netdev@vger.kernel.org; e1000-devel@lists.sourceforge.net
>Subject: RE: ixgbe: macvlan on PF/VF when SRIOV is enabled
>
>On Mon, 2010-05-24 at 10:54 -0700, Rose, Gregory V wrote:
>> We look forward to it and will be happy to provide feedback.
>I have submitted the patch to make macvlan on PF works when SRIOV is
>enabled.
>
>> One thing you can do is allocate VFs and then load the VF driver in
>> your host domain and then assign each of them a macvlan filter.  You'd
>> get a similar effect.
>
>That's I am trying to make it work for macvlan on VFs in host domain. I
>need to add VF secondary addresses in address filter, right?

Just use ifconfig and vconfig utilities to set the MAC and VLAN for each VF.  There shouldn't be any need for secondary addresses because they're not like physical devices where each VF has a pre-programmed HW MAC address.  The initial MAC address of each VF is generated on the fly during the PF driver initialization. You can change it as you see fit and then put the VF on a VLAN using vconfig.  After you do that you have a macvlan filter for that VF.

>
>Do you have any aggregation performance comparison between multiple
>macvlans on PF and single macvlan per VF in host domain?

No, we haven't done any performance testing of that particular sort but I imagine you should be able to get close to line rates using multiple VFs with a single macvlan filter for each.

Regards,

- Greg

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Shirley Ma]

On Tue, 2010-05-25 at 08:57 -0700, Rose, Gregory V wrote:
> Just use ifconfig and vconfig utilities to set the MAC and VLAN for
> each VF.  There shouldn't be any need for secondary addresses because
> they're not like physical devices where each VF has a pre-programmed
> HW MAC address.  The initial MAC address of each VF is generated on
> the fly during the PF driver initialization. You can change it as you
> see fit and then put the VF on a VLAN using vconfig.  After you do
> that you have a macvlan filter for that VF.

I run macvlan test not vlan. macvlan is used to give a second MAC
address to a network adapter and see it as a new device at the higher
levels. The command is used as follow:

ip link add link eth4 address 54:52:00:35:e3:20 macvlan2 type macvlan

It will create an interface name macvlan2 with above MAC address. In
kernel, netdev eth4 maintains this secondary address.

Thanks
Shirley

RE: ixgbe: macvlan on PF/VF when SRIOV is enabled

[Rose, Gregory V]

>-----Original Message-----
>From: Shirley Ma [mailto:mashirle@us.ibm.com]
>Sent: Tuesday, May 25, 2010 9:51 AM
>To: Rose, Gregory V
>Cc: Kirsher, Jeffrey T; davem@davemloft.net; kvm@vger.kernel.org;
>netdev@vger.kernel.org; e1000-devel@lists.sourceforge.net
>Subject: RE: ixgbe: macvlan on PF/VF when SRIOV is enabled
>
>On Tue, 2010-05-25 at 08:57 -0700, Rose, Gregory V wrote:
>> Just use ifconfig and vconfig utilities to set the MAC and VLAN for
>> each VF.  There shouldn't be any need for secondary addresses because
>> they're not like physical devices where each VF has a pre-programmed
>> HW MAC address.  The initial MAC address of each VF is generated on
>> the fly during the PF driver initialization. You can change it as you
>> see fit and then put the VF on a VLAN using vconfig.  After you do
>> that you have a macvlan filter for that VF.
>
>I run macvlan test not vlan. macvlan is used to give a second MAC
>address to a network adapter and see it as a new device at the higher
>levels. The command is used as follow:
>
>ip link add link eth4 address 54:52:00:35:e3:20 macvlan2 type macvlan
>
>It will create an interface name macvlan2 with above MAC address. In
>kernel, netdev eth4 maintains this secondary address.

Right, so what I'm saying is that if you load the VF driver it will create and <eth> interface for each VF you've created.  You can assign a MAC and VLAN to that <eth> interface and will get essentially the same behavior.

- Greg