Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders <sven.anders@anduras.de>]

Hello Dave,

I want so report back that I solved the problem.
The problem was not in the i40e driver.

Explanation:
We upgraded a server with the new 10G (quad) network card.
The server was working with kernel in version 4.14.238.
Due to the new network card, we were forced to upgrade to a newer kernel and we used 5.15.38.
The network configuration stayed the same, but we had the reported problem.

Due to a mysterious fix with an older version of the kernel, there was a part of
the network configuration that set the mac address on all vlans.
This was the cause of the problem. The older kernel ignored this settings but
the newer kernel performed (correctly) some actions on it (and maybe sent some instructions
to the network card). Forcing the promiscous mode by triggering the error must disabled some
previous settings of the network card and made it working again.

Nevertheless, thanks for your time testing it. It helped me to eliminate the driver as the
cause and forced me to dig deeper on my side.
Removing the part, which was setting the mac address, elimiated the errounous behaviour
and the network card is not working.

Regards
   Sven

Am 21.06.22 um 23:18 schrieb Switzer, David:
> 
>> -----Original Message-----
>> From: Switzer, David
>> Sent: Thursday, June 16, 2022 1:31 PM
>> To: Sven Anders <sven.anders@anduras.de>; intel-wired-lan@osuosl.org
>> Subject: RE: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>
>>> Hallo Dave,
>> Hi Sven,
> 
> Hello Sven!
>>
>>>
>>> thanks for your answer.
>> You're very welcome!
>>
>>>
>>> I tried to disable the LLDP engine, as suggested, but it did not change
>>> anything.
>>>
>>> To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
>>> which works with ARP negotiation.
>>> If I'm using the "active-backup" bonding mode, it all works fine. But
>>> this is nearly the same as using only one interface, which works too.
>>>
>>> In my test setup, I'm using two "HPE 2530-48G-PoE+" switches
>>> (YA.16.10.0002), each is connected with one cable to the network card.
>>> My test is very simple:
>>> After configuring the bond, I used a continous ping to the Linux server
>>> and I see many drops. If I toggle the "ntuples support", the drops disappear.
>>>
>>> Can we compare the settings of your setup with mine? Maybe you missed
>>> something...
>> Just realized we missed the 2nd switch(we're working on setting that up now)
> I got my 2nd switch configured with 36 vlans on the bond, I still can't reproduce the issue.
> 
> I'm in a test environment so I don't have much of a "network" between the two switches.  I was wondering how your switches were configured and if there was anything else going on with  the network(people streaming, backups occurring, etc).
> 
>>
>>> How can I assist further?
>> Can you enable source pruning on the ports that are part of your bond?  There
>> might be some confusion in the fw/sw with the two interfaces and packets
>> are getting ignored:
>> ethtool --set-priv-flags <interface name> disable-source-pruning on
>>
> I was reading over this and realized I contradicted myself above.  I would like to see if disabling source pruning would help.
> 
> I hope you're having a good week!
> Dave
> 
>> Let me know if that helps.  It sounds like we'll be getting our two switches
>> later today so we can make sure we have our environment matching yours.
>>
>> Have a great day!
>> Dave
>>
>>
>>>
>>> Regards
>>>   Sven
>>>
>>> Am 11.06.22 um 03:27 schrieb Switzer, David:
>>>>> -----Original Message-----
>>>>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
>>>>> Of Sven Anders
>>>>> Sent: Monday, June 6, 2022 11:05 PM
>>>>> To: intel-wired-lan@osuosl.org
>>>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710
>>>>> hardware
>>>>>
>>>>> Hello!
>>>> Hi Sven!  My apologies on taking so long to reply.
>>>>>
>>>>> Was anybody able to reproduce the problem?
>>>> I haven't been able to reproduce your issue yet, but with your
>>>> description it sounds that the issue could be related to the X710
>>>> hardware LLDP engine consuming the LLDP packets.  To disable the
>>> hardware engine, user the following command on each of the ports that
>>> you're using in the bond:
>>>>
>>>> ethtool --set-priv-flags <interface name> disable-fw-lldp on
>>>>
>>>> Then the LLDP packets that the bonding modules relies on will be
>>>> forwarded
>>> by the hardware to the OS network stack.
>>>>
>>>> I hope you have a great weekend!
>>>> Dave
>>>>
>>>>> Do you need assistance or further information?
>>>>>
>>>>> Regards
>>>>>    Sven
>>>>>
>>>>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> This is a follow up to my question. I did not hear anything so
>>>>>>>>> far, but I tried to find some some information meanwhile.
>>>>>>>>>
>>>>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode
>>>>>>>>> forced on" maybe triggered, because I'm hitting a limit here.
>>>>>>>
>>>>>>> Yes, typically this is a response from our firmware that a table
>>>>>>> is full in hardware, and I'm guessing that you might be running
>>>>>>> into a filter
>>>>> limit due to using vlans?
>>>>>>
>>>>>> That's what I suspect, but I did know the hardware and firmware
>>>>>> insufficiently to be sure.
>>>>>> What I'm wondering is: Why this is only triggered if I toggle the
>>>>>> "ntuples
>>>>> support"
>>>>>> and not earlier when setting up the interface?
>>>>>>
>>>>>>>>> Data packets are only received sporadically. If I run the same
>>>>>>>>> test with only one slave port, it works without problems.
>>>>>>>
>>>>>>> And there are no counters going up in ethtool -S when you
>>>>>>> receive/drop
>>>>> packets?
>>>>>>
>>>>>> I'm not sure here. I tried to inspect the counters and it seems
>>>>>> that the counters are going up slower in this case, but it's
>>>>>> difficult to say, because there is some "noise" on the line and I
>>>>>> do not have direct access to the hardware at the moment.
>>>>>>
>>>>>> If you need any further information or tests, just contact me.
>>>>>
>>>>> Regards
>>>>>      Sven Anders

Regards
   Sven Anders

-- 
   Sven Anders                  () UTF-8 Ribbon Campaign
                                                   /\ Support plain text e-mail
   ANDURAS intranet security AG
   Messestrasse 3 - 94036 Passau - Germany
   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
    - Benjamin Franklin

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan