[Intel-wired-lan] i40e MDD events

[Intel-wired-lan] i40e MDD events

[Dan Streetman]

Hi, I'm from Canonical support, and I have reports from an Ubuntu user
seeing MDD events occurring on a i40e nic, from the PF (not a VF)
while using the Ubuntu 4.4 kernel.  I found two commits that I thought
addressed this problem, but they did not help - the reporter still is
seeing MDD events, which reset the PF for every MDD event.  The
commits I thought fixed this, which have been added but still don't
fix the MDD events, are:

commit 3f3f7cb875c0f621485644d4fd7453b0d37f00e4
Author: Alexander Duyck <aduyck@mirantis.com>
Date:   Wed Mar 30 16:15:37 2016 -0700

    i40e/i40evf: Limit TSO to 7 descriptors for payload instead of 8 per packet


and then:

commit 841493a3f64395b60554afbcaa17f4350f90e764
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Tue Sep 6 18:05:04 2016 -0700

    i40e: Limit TX descriptor count in cases where frag size is greater than 16K


The events are always of the form:
i40e 0000:0c:00.1: Malicious Driver Detection event 0x02 on TX queue
35 PF number 0x01 VF number 0x00

the queue and PF may vary, but it's always from the PF (never from VF)
and the event is always 2.

Unfortunately, the i40e spec doesn't really provide much helpful info
about what might specifically cause a MDD event, or what the event
codes mean (at least, not that I've found in the datasheet).  Does
anyone know what MDD event "2" means, does that indicate too many TX
descriptors, as was the issue for the above commits?  Or something
else?

Are there any other MDD-related commits that I missed since those 2
were added?  It's quite frustrating, because normally errors indicate
what the problem was, but the MDD errors have absolutely no indication
of what the NIC didn't like.

Thanks!

[Intel-wired-lan] i40e MDD events

[Wyborny, Carolyn]

> -----Original Message-----
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On
> Behalf Of Dan Streetman
> Sent: Thursday, September 28, 2017 1:53 PM
> To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; Duyck, Alexander H
> <alexander.h.duyck@intel.com>
> Subject: [Intel-wired-lan] i40e MDD events
> 
> Hi, I'm from Canonical support, and I have reports from an Ubuntu user
> seeing MDD events occurring on a i40e nic, from the PF (not a VF)
> while using the Ubuntu 4.4 kernel.  I found two commits that I thought
> addressed this problem, but they did not help - the reporter still is
> seeing MDD events, which reset the PF for every MDD event.  The
> commits I thought fixed this, which have been added but still don't
> fix the MDD events, are:
> 

Hello Dan,

What is the firmware version on the device?  ethtool -i output or dmesg log should provide it.  I believe the device needs a fw update.   They are available  on our download center here:  https://downloadcenter.intel.com/download/24769
A fw update will usually require a driver update as well, but the fw version info from the user will help with this determination.

Thanks,

Carolyn

Carolyn Wyborny 
Linux Development 
Networking Division 
Intel Corporation 

[Intel-wired-lan] i40e MDD events

[Dan Streetman]

On Thu, Sep 28, 2017 at 7:27 PM, Wyborny, Carolyn
<carolyn.wyborny@intel.com> wrote:
>> -----Original Message-----
>> From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On
>> Behalf Of Dan Streetman
>> Sent: Thursday, September 28, 2017 1:53 PM
>> To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T
>> <jeffrey.t.kirsher@intel.com>; Duyck, Alexander H
>> <alexander.h.duyck@intel.com>
>> Subject: [Intel-wired-lan] i40e MDD events
>>
>> Hi, I'm from Canonical support, and I have reports from an Ubuntu user
>> seeing MDD events occurring on a i40e nic, from the PF (not a VF)
>> while using the Ubuntu 4.4 kernel.  I found two commits that I thought
>> addressed this problem, but they did not help - the reporter still is
>> seeing MDD events, which reset the PF for every MDD event.  The
>> commits I thought fixed this, which have been added but still don't
>> fix the MDD events, are:
>>
>
> Hello Dan,
>
> What is the firmware version on the device?

it's running 5.05

>  ethtool -i output or dmesg log should provide it.  I believe the device needs a fw update.   They are available  on our download center here:  https://downloadcenter.intel.com/download/24769
> A fw update will usually require a driver update as well, but the fw version info from the user will help with this determination.

I see you just added a new 6.01 fw a little over a month ago, is there
a known fw problem related to this that 6.01 fixes?  I'll ask them to
update anyway, to see if it helps.

Can you answer my questions about specifically what events in the NIC
will cause it to generate a MDD, and specifically what does MDD event
"2" indicate?

>
> Thanks,
>
> Carolyn
>
> Carolyn Wyborny
> Linux Development
> Networking Division
> Intel Corporation
>
>

[Intel-wired-lan] i40e MDD events

[Dan Streetman]

On Fri, Sep 29, 2017 at 8:17 AM, Dan Streetman
<dan.streetman@canonical.com> wrote:
> On Thu, Sep 28, 2017 at 7:27 PM, Wyborny, Carolyn
> <carolyn.wyborny@intel.com> wrote:
>>> -----Original Message-----
>>> From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On
>>> Behalf Of Dan Streetman
>>> Sent: Thursday, September 28, 2017 1:53 PM
>>> To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T
>>> <jeffrey.t.kirsher@intel.com>; Duyck, Alexander H
>>> <alexander.h.duyck@intel.com>
>>> Subject: [Intel-wired-lan] i40e MDD events
>>>
>>> Hi, I'm from Canonical support, and I have reports from an Ubuntu user
>>> seeing MDD events occurring on a i40e nic, from the PF (not a VF)
>>> while using the Ubuntu 4.4 kernel.  I found two commits that I thought
>>> addressed this problem, but they did not help - the reporter still is
>>> seeing MDD events, which reset the PF for every MDD event.  The
>>> commits I thought fixed this, which have been added but still don't
>>> fix the MDD events, are:
>>>
>>
>> Hello Dan,
>>
>> What is the firmware version on the device?
>
> it's running 5.05
>
>>  ethtool -i output or dmesg log should provide it.  I believe the device needs a fw update.   They are available  on our download center here:  https://downloadcenter.intel.com/download/24769
>> A fw update will usually require a driver update as well, but the fw version info from the user will help with this determination.
>
> I see you just added a new 6.01 fw a little over a month ago, is there
> a known fw problem related to this that 6.01 fixes?  I'll ask them to
> update anyway, to see if it helps.

unfortunately the updated firmware did not help; an MDD event happened
again after upgrading to 6.01.

>
> Can you answer my questions about specifically what events in the NIC
> will cause it to generate a MDD, and specifically what does MDD event
> "2" indicate?

It there any possibility to get more info about the MDD event details?
 Like what specifically the NIC didn't like?

>
>>
>> Thanks,
>>
>> Carolyn
>>
>> Carolyn Wyborny
>> Linux Development
>> Networking Division
>> Intel Corporation
>>
>>

[Intel-wired-lan] i40e MDD events

[Jesse Brandeburg]

On Mon, 2 Oct 2017 09:31:46 -0400
Dan Streetman <dan.streetman@canonical.com> wrote:

> unfortunately the updated firmware did not help; an MDD event happened
> again after upgrading to 6.01.

Hi Dan, sorry that didn't help.

> > Can you answer my questions about specifically what events in the NIC
> > will cause it to generate a MDD, and specifically what does MDD event
> > "2" indicate?  
> 
> It there any possibility to get more info about the MDD event details?
>  Like what specifically the NIC didn't like?

Unfortunately the MDD mechanism didn't work for reporting as we had
hoped.  I'll send you the troubleshooting instructions off-list.

Jesse