Re: ixgbe : query regarding your code changes for VF mac add

[Ivan Boule <ivan.boule@6wind.com>]

Hi Santosh,

Things are a little bit more complex.
When the permanent MAC address of a VF is assigned by the PF driver, it 
cannot be changed nor deleted by the VF through commands issued to the 
PF driver. In this case, the function xgbevf_remove_mac_addr() must not 
add the permanent MAC address again, as currently done.

Conversely, when the permanent MAC address of a VF is assigned by the Vf 
itself, then it will be also deleted by the call to 
ixgbevf_set_uc_addr_vf(hw, 0, NULL) in the function 
xgbevf_remove_mac_addr (as other MAC addresses that have been previously 
added through a call to the function ixgbevf_add_mac_addr).
In this case, the function xgbevf_remove_mac_addr must effectively add 
the permanent MAC address again.

To resume, the ixgbevf PMD must record in a new dedicated flag if the 
permanent MAC address of a VF has been assigned by the PF driver or not, 
and test this flag in the function xgbevf_remove_mac_addr to decide if 
it must add again the permanent MAC address of the VF.

Regards,
Ivan

On 04/26/2016 07:36 AM, santosh wrote:
> Hi
>
> Looks like there is a bug in API "ixgbevf_remove_mac_addr()".   For
> deleting a given MAC it deletes all MAC (including permanent MAC) and
> while adding (it does after few statements), it skips to add permanent
> MAC.
>
> ixgbevf_remove_mac_addr(struct rte_eth_dev *dev, uint32_t index)
> {
> .......
>
>      /*
>       * The IXGBE_VF_SET_MACVLAN command of the ixgbe-pf driver does
>       * not support the deletion of a given MAC address.
>       * Instead, it imposes to delete all MAC addresses, then to add again
>       * all MAC addresses with the exception of the one to be deleted.
>       */
>
>
>      (void) ixgbevf_set_uc_addr_vf(hw, 0, NULL);
> ...
>
> 2.   Following modification at "ixgbevf_remove_mac_addr()}  also
> helped to make ping works
>
> $ git diff  dpdk/lib/librte_pmd_ixgbe/ixgbe_ethdev.c
> --- a/dpdk/lib/librte_pmd_ixgbe/ixgbe_ethdev.c
> +++ b/dpdk/lib/librte_pmd_ixgbe/ixgbe_ethdev.c
>
>           @@ -3530,10 +3557,7 @@ ixgbevf_remove_mac_addr(struct
> rte_eth_dev *dev, uint32_t index)
>                  /* Skip NULL MAC addresses */
>                  if (is_zero_ether_addr(mac_addr))
>                          continue;
> -               /* Skip the permanent MAC address */
> -               if (memcmp(perm_addr, mac_addr, sizeof(struct ether_addr)) == 0)
> -                       continue;
>
> On Mon, Apr 25, 2016 at 7:05 PM, santosh <santosh.iitg@gmail.com> wrote:
>> Hi Ivan
>>
>>   ixgbevf_set_default_mac_addr()   could not find in our code base.
>> put traces at other places as suggested by you.
>>   Log at "eth_ixgbevf_dev_init"  never displayed
>> Rest logs displayed as shown below.  I re-build the  driver module and
>> loaded on our virtual router and rebooted the system.
>>
>> 1.
>> Logs at the time of boot up:
>> -----------------------------------------
>>
>> INIT: Initializing NIC port 0 RX queue 0 ...
>> INIT: Initializing NIC port 0 TX queue 0 ...
>> Santosh ixgbevf_add_mac_addr portid=0 mac=00:50:56:A0:10:C2
>> .....
>> ....
>>
>> Santosh ixgbevf_add_mac_addr portid=0 mac=00:50:56:A0:10:C2
>> RUNTIME: Detected port  0 status changed to UP.
>>
>> 2.
>> a.
>> Configured  a new MAC at Virtual Router CLI
>>
>> root@mx86-bgl-2-r1# set interfaces ge-0/0/0 mac 00:50:56:a0:a0:c3
>>
>> [edit]
>> root@mx86-bgl-2-r1# commit
>> commit complete
>>
>> [edit]
>> root@mx86-bgl-2-r1# show interfaces
>> ge-0/0/0 {
>>      mac 00:50:56:a0:a0:c3;
>>      unit 0 {
>>          family inet {
>>              address 10.1.1.102/24;
>>          }
>>      }
>> }
>>
>> [edit]
>> root@mx86-bgl-2-r1# run ping 10.1.1.101
>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>> 64 bytes from 10.1.1.101: icmp_seq=0 ttl=64 time=2.142 ms
>> 64 bytes from 10.1.1.101: icmp_seq=1 ttl=64 time=8.465 ms
>> ^C
>> --- 10.1.1.101 ping statistics ---
>> 2 packets transmitted, 2 packets received, 0% packet loss
>> round-trip min/avg/max/stddev = 2.142/5.303/8.465/3.162 ms
>>
>> [edit]
>> root@mx86-bgl-2-r1#
>>
>> b.
>>     Logs at console for above config:
>>     -------------------------------------------------
>>
>> root@localhost:/home/pfe# Santosh ixgbevf_add_mac_addr portid=0
>> mac=00:50:56:A0:A0:C3
>>
>> root@localhost:/home/pfe#
>>
>>
>> 3.
>> a. Deleted above config MAC
>> root@mx86-bgl-2-r1# delete interfaces ge-0/0/0 mac
>>
>> [edit]
>> root@mx86-bgl-2-r1# commit
>> commit complete
>>
>> [edit]
>> root@mx86-bgl-2-r1# show interfaces
>> ge-0/0/0 {
>>      unit 0 {
>>          family inet {
>>              address 10.1.1.102/24;
>>          }
>>      }
>> }
>>
>> [edit]
>> root@mx86-bgl-2-r1# run ping 10.1.1.101
>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>> ^C
>> --- 10.1.1.101 ping statistics ---
>> 3 packets transmitted, 0 packets received, 100% packet loss
>>
>> [edit]
>> root@mx86-bgl-2-r1#
>>
>> b.
>>   Logs at console for above cofig
>>   -------------------------
>>
>>
>> root@localhost:/home/pfe# ixgbevf_remove_mac_addr santosh portid=0
>> index=1 mac=00:50:56:A0:A0:C3
>>
>>
>>
>> Thanks
>> Santosh
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Apr 22, 2016 at 12:51 PM, Ivan Boule <ivan.boule@6wind.com> wrote:
>>> Hi Santosh,
>>>
>>> My job at 6WIND does not consist in answering to DPDK questions. In general,
>>> I have other priorities, including vacations...
>>> In the meantime, nobody prevents you to add traces in the code to really
>>> understand what happens, as suggested in my last answer.
>>>
>>> Regards,
>>> Ivan
>>>
>>>
>>> On 04/21/2016 07:55 AM, santosh wrote:
>>>>
>>>> Hi Ivan and team,
>>>>
>>>> Please respond to my last mail and  let me know if there is any
>>>> alternate way to handle this.
>>>> Our release is in pending due to this issue.
>>>>
>>>>
>>>> Thanks & Regards
>>>> Santosh
>>>>
>>>> On Wed, Apr 20, 2016 at 2:35 PM, santosh <santosh.iitg@gmail.com> wrote:
>>>>>
>>>>> Hi Ivan,
>>>>>
>>>>> Thanks for your response.
>>>>>
>>>>> Let me explain you the issue that we are facing on our virtual router
>>>>> in VMware environment.
>>>>>
>>>>> 1. We are using ixgbe driver , SRIOV enabled .
>>>>>       root@localhost:~# lspci
>>>>>         .... "Intel Corporation 82599 Ethernet Controller Virtual
>>>>> Function"
>>>>>
>>>>> 2.  "mx86-bgl-1-r1"  is our router under testing and  R2 is a standard
>>>>> router.
>>>>>
>>>>>         mx86-bgl-1-r1 is connected to R2
>>>>>         mx86-bgl-1-r1 (10.1.1.103)------------------>R2(10.1.1.101)
>>>>>
>>>>> 2. This time ping test passes
>>>>>
>>>>> root@mx86-bgl-1-r1# show interfaces
>>>>> ge-0/0/0 {
>>>>>       unit 0 {
>>>>>           family inet {
>>>>>               address 10.1.1.103/24;
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1#
>>>>> root@mx86-bgl-1-r1# run ping 10.1.1.101
>>>>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>>>>> 64 bytes from 10.1.1.101: icmp_seq=0 ttl=64 time=0.980 ms
>>>>> 64 bytes from 10.1.1.101: icmp_seq=1 ttl=64 time=1.433 ms
>>>>>
>>>>>
>>>>> 3.  Default MAC address of interface ge-0/0/0  is as shown below:
>>>>>
>>>>> root@mx86-bgl-1-r1# run show interfaces ge-0/0/0 extensive | grep current
>>>>>     Current address: 02:06:0a:0e:ff:f0, Hardware address:
>>>>> 02:06:0a:0e:ff:f0
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1#
>>>>>
>>>>>
>>>>> 4.  Now I am changing MAC. Ping works this time also
>>>>>
>>>>> root@mx86-bgl-1-r1# set interfaces ge-0/0/0 mac 02:06:0a:0a:ff:f0
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# commit
>>>>> commit complete
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# run show interfaces ge-0/0/0 extensive | grep current
>>>>>     Current address: 02:06:0a:0a:ff:f0, Hardware address:
>>>>> 02:06:0a:0e:ff:f0
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# show interfaces
>>>>> ge-0/0/0 {
>>>>>       mac 02:06:0a:0a:ff:f0;
>>>>>       unit 0 {
>>>>>           family inet {
>>>>>               address 10.1.1.103/24;
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1#
>>>>> root@mx86-bgl-1-r1# run ping 10.1.1.101
>>>>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>>>>> 64 bytes from 10.1.1.101: icmp_seq=0 ttl=64 time=1.338 ms
>>>>> 64 bytes from 10.1.1.101: icmp_seq=1 ttl=64 time=8.832 ms
>>>>> 64 bytes from 10.1.1.101: icmp_seq=2 ttl=64 time=1.292 ms
>>>>>
>>>>>
>>>>> 5.  I am deleting the above configured MAC.
>>>>>        In this case newly configured MAC as shown above will be deleted
>>>>> and Default MAC (02:06:0a:0e:ff:f0)
>>>>>         will be applied.  Ping fails at this step.  "return statement
>>>>> added by you is not allowing to configure default MAC.
>>>>>
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# delete interfaces ge-0/0/0 mac
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# commit
>>>>> commit complete
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# show interfaces
>>>>> ge-0/0/0 {
>>>>>       unit 0 {
>>>>>           family inet {
>>>>>               address 10.1.1.103/24;
>>>>>           }
>>>>>       }
>>>>> }
>>>>>
>>>>> [edit]
>>>>>
>>>>> root@mx86-bgl-1-r1# run show interfaces ge-0/0/0 extensive | grep
>>>>> current     // Displays value from router's local database not from
>>>>> NIC.
>>>>>     Current address: 02:06:0a:0e:ff:f0, Hardware address:
>>>>> 02:06:0a:0e:ff:f0
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1# run ping 10.1.1.101
>>>>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>>>>> ^C
>>>>> 2 packets transmitted, 0 packets received, 100% packet loss
>>>>>
>>>>> [edit]
>>>>> root@mx86-bgl-1-r1#
>>>>>
>>>>>
>>>>> 7. LOGs:    (I have added a debug log just before the return statement
>>>>> that you place)
>>>>>
>>>>> Log o/p in failure case
>>>>> ....
>>>>> Santosh ixgbevf_add_mac_addr returning
>>>>> ....
>>>>>
>>>>> code changes:
>>>>> -----------------------------
>>>>> ixgbevf_add_mac_addr(....)  {
>>>>> ...
>>>>>       if (memcmp(hw->mac.perm_addr, mac_addr, sizeof(struct ether_addr))
>>>>> == 0) {
>>>>>           PMD_DRV_LOG(DEBUG, "Existing MAC \n");
>>>>>           printf("Santosh %s returning \n",__FUNCTION__);
>>>>>           return;
>>>>>       }
>>>>>
>>>>>
>>>>> 8.  If I remove the above "return" statement, build the driver, loaded
>>>>> in router and repeat the steps-2 to steps-5 of MAC add and MAC delete
>>>>> on our router then ping passes.
>>>>>
>>>>> root@mx86-bgl-1-r1# run ping 10.1.1.101
>>>>> PING 10.1.1.101 (10.1.1.101): 56 data bytes
>>>>> 64 bytes from 10.1.1.101: icmp_seq=0 ttl=64 time=2.356 ms
>>>>> 64 bytes from 10.1.1.101: icmp_seq=1 ttl=64 time=1.415 ms
>>>>> 64 bytes from 10.1.1.101: icmp_seq=2 ttl=64 time=1.226 ms
>>>>> ^C
>>>>> --- 10.1.1.101 ping statistics ---
>>>>> 3 packets transmitted, 3 packets received, 0% packet loss
>>>>> round-trip min/avg/max/stddev = 1.226/1.666/2.356/0.494 ms
>>>>>
>>>>>
>>>>> 9.  Please let me know whether is it  fine to remove that return
>>>>> statement added by you in  "ixgbevf_add_mac_addr"  ?
>>>>>
>>>>>
>>>>>
>>>>> Thanks & Regards,
>>>>> Santosh
>>>>>
>>>>> On Wed, Apr 20, 2016 at 3:31 AM, Ivan Boule <ivan.boule@6wind.com> wrote:
>>>>>>
>>>>>> Hi Santosh,
>>>>>>
>>>>>> I do not get exactly what you attempt to do on a VF.
>>>>>> Are you first deleting the so-called permanent MAC address by a call to
>>>>>> the
>>>>>> function ixgbevf_remove_mac_addr() ? This operation is not allowed.
>>>>>> Can you explain exactly the sequence of operations that are done, so
>>>>>> that I
>>>>>> can understand how the test (memcmp(hw->mac.perm_addr, mac_addr,
>>>>>> sizeof(struct ether_addr)) == 0) in the function ixgbevf_add_mac_addr()
>>>>>> prevents them to be successfully performed.
>>>>>>
>>>>>> Ivan
>>>>>>
>>>>>> PS : please, can you CC your emails to dev@dpdk.org
>>>>>>
>>>>>>
>>>>>> 2016-04-19 17:01 GMT+02:00 santosh <santosh.iitg@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>> Hi Ivan,
>>>>>>>
>>>>>>> Your following code changes causing issue in Vmware environment.
>>>>>>>
>>>>>>> ----------------------------------- -------------------
>>>>>>> ------------------------------
>>>>>>> + /*
>>>>>>> + * On a 82599 VF, adding again the same MAC addr is not an idempotent
>>>>>>> + * operation. Trap this case to avoid exhausting the [very limited]
>>>>>>> + * set of PF resources used to store VF MAC addresses.
>>>>>>> + */
>>>>>>> + if (memcmp(hw->mac.perm_addr, mac_addr, sizeof(struct ether_addr)) ==
>>>>>>> 0)
>>>>>>> + return;
>>>>>>>     diag = ixgbevf_set_uc_addr_vf(hw, 2, mac_addr->addr_bytes);
>>>>>>> --------------------------------------------------------------------
>>>>>>> ------------- -------
>>>>>>>
>>>>>>>
>>>>>>> Issue:
>>>>>>> At CLI , we have provision to add /del MAC of an interface.
>>>>>>> During MAC delete, existing MAC is deleted and default MAC is applied.
>>>>>>> This default MAC is not being applied in VMware environment
>>>>>>> successfully due to "return" statement
>>>>>>> in your above code changes. As a result traffic is stopped completely.
>>>>>>> If I remove above
>>>>>>> "return" statement then traffic continues to flow after MAC delete.
>>>>>>>
>>>>>>> Please let me know your suggestion to handle this scenario .  If I
>>>>>>> remove "return" what will be the consequences ?
>>>>>>>
>>>>>>> If removing "return" statement is not good idea then what are other
>>>>>>> way to handle MAC delete scenario ?  we have only 1 VF per PF in our
>>>>>>> setup as of now.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Santosh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ivan BOULE
>>>>>> 6WIND
>>>>>> Software Engineer
>>>>>>
>>>>>> Tel: +33 1 39 30 92 47
>>>>>> Mob: + 33 6 77 25 26 38
>>>>>> Fax: +33 1 39 30 92 11
>>>>>> ivan.boule@6wind.com
>>>>>> www.6wind.com
>>>>>> Join the Multicore Packet Processing Forum:
>>>>>> www.multicorepacketprocessing.com
>>>>>>
>>>>>> Ce courriel ainsi que toutes les pièces jointes, est uniquement destiné
>>>>>> à
>>>>>> son ou ses destinataires. Il contient des informations confidentielles
>>>>>> qui
>>>>>> sont la propriété de 6WIND. Toute révélation, distribution ou copie des
>>>>>> informations qu'il contient est strictement interdite. Si vous avez reçu
>>>>>> ce
>>>>>> message par erreur, veuillez immédiatement le signaler à l'émetteur et
>>>>>> détruire toutes les données reçues.
>>>>>>
>>>>>> This e-mail message, including any attachments, is for the sole use of
>>>>>> the
>>>>>> intended recipient(s) and contains information that is confidential and
>>>>>> proprietary to 6WIND. All unauthorized review, use, disclosure or
>>>>>> distribution is prohibited. If you are not the intended recipient,
>>>>>> please
>>>>>> contact the sender by reply e-mail and destroy all copies of the
>>>>>> original
>>>>>> message.
>>>>>>
>>>
>>>
>>> --
>>> Ivan Boule
>>> 6WIND Development Engineer


-- 
Ivan Boule
6WIND Development Engineer