[Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

Hi,

My "Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)" device stopped working on recent kernel builds (fails to show up in 'ifconfig' or to get an IP address).  I bisected the failure down to the following commit:

commit a0285236ab93fdfdd1008afaa04561d142d6c276 (HEAD, refs/bisect/bad)
Author: Jedrzej Jagielski <jedrzej.jagielski@intel.com>
Date:   Thu Apr 10 14:59:56 2025 +0200

    ixgbe: add initial devlink support

    Add an initial support for devlink interface to ixgbe driver.

    Similarly to i40e driver the implementation doesn't enable
    devlink to manage device-wide configuration. Devlink instance
    is created for each physical function of PCIe device.

    Create separate directory for devlink related ixgbe files
    and use naming scheme similar to the one used in the ice driver.

    Add a stub for Documentation, to be extended by further patches.

    Change struct ixgbe_adapter allocation to be done by devlink (Przemek),
    as suggested by Jiri.

    Reviewed-by: Mateusz Polchlopek <mateusz.polchlopek@intel.com>
    Co-developed-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
    Signed-off-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
    Tested-by: Bharath R <bharath.r@intel.com>
    Signed-off-by: Jedrzej Jagielski <jedrzej.jagielski@intel.com>
    Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>

Normally, the device is given a name like enp194s0f0 and connects to the wired network.  Starting with this patch, it fails to do so.  I'm not familiar with how to debug this particular device, so please let me know what other information would be of assistance.

lspci -vv information (on a working build):

c2:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
        Subsystem: ASRock Incorporation Ethernet Controller 10G X550T
        Physical Slot: 0-1
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 77
        IOMMU group: 12
        Region 0: Memory at 18020800000 (64-bit, prefetchable) [size=4M]
        Region 4: Memory at 18020c04000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at b6000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

Thanks --David Kaplan

Re: [Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Przemek Kitszel]

On 6/16/25 21:42, Kaplan, David wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
> Hi,
> 
> My "Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)" device stopped working on recent kernel builds (fails to show up in 'ifconfig' or to get an IP address).  I bisected the failure down to the following commit:
> 
> commit a0285236ab93fdfdd1008afaa04561d142d6c276 (HEAD, refs/bisect/bad)
> Author: Jedrzej Jagielski <jedrzej.jagielski@intel.com>
> Date:   Thu Apr 10 14:59:56 2025 +0200
> 
>      ixgbe: add initial devlink support

[..]

> Normally, the device is given a name like enp194s0f0 and connects to the wired network.  

now the name is likely to be different, please see this thread:
https://lkml.org/lkml/2025/4/24/1750

Is it possible that your network setup script has some part of interface
discovery hardcoded?

> Starting with this patch, it fails to do so.  I'm not familiar with how to debug this particular device, so please let me know what other information would be of assistance.
> 
> lspci -vv information (on a working build):
> 
> c2:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
>          Subsystem: ASRock Incorporation Ethernet Controller 10G X550T
>          Physical Slot: 0-1
>          Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>          Latency: 0
>          Interrupt: pin A routed to IRQ 77
>          IOMMU group: 12
>          Region 0: Memory at 18020800000 (64-bit, prefetchable) [size=4M]
>          Region 4: Memory at 18020c04000 (64-bit, prefetchable) [size=16K]
>          Expansion ROM at b6000000 [disabled] [size=512K]
>          Capabilities: <access denied>
>          Kernel driver in use: ixgbe
>          Kernel modules: ixgbe
> 
> Thanks --David Kaplan
> 
> 

Re: [Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Przemek Kitszel <przemyslaw.kitszel@intel.com>
> Sent: Tuesday, June 17, 2025 4:01 AM
> To: Kaplan, David <David.Kaplan@amd.com>; jedrzej.jagielski@intel.com
> Cc: anthony.l.nguyen@intel.com; netdev@vger.kernel.org; horms@kernel.org;
> Mateusz Polchlopek <mateusz.polchlopek@intel.com>; Bharath R
> <bharath.r@intel.com>; regressions@lists.linux.dev; intel-wired-
> lan@lists.osuosl.org
> Subject: Re: [REGRESSION] ixgbe broken after devlink support patch
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 6/16/25 21:42, Kaplan, David wrote:
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> > Hi,
> >
> > My "Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)"
> device stopped working on recent kernel builds (fails to show up in 'ifconfig' or to get
> an IP address).  I bisected the failure down to the following commit:
> >
> > commit a0285236ab93fdfdd1008afaa04561d142d6c276 (HEAD, refs/bisect/bad)
> > Author: Jedrzej Jagielski <jedrzej.jagielski@intel.com>
> > Date:   Thu Apr 10 14:59:56 2025 +0200
> >
> >      ixgbe: add initial devlink support
>
> [..]
>
> > Normally, the device is given a name like enp194s0f0 and connects to the wired
> network.
>
> now the name is likely to be different, please see this thread:
> https://lkml.org/lkml/2025/4/24/1750
>
> Is it possible that your network setup script has some part of interface
> discovery hardcoded?
>

Thanks for the pointer, that seemed to be the issue.  I was running Ubuntu 22.04.5 LTS and I had to edit /etc/netplan/01-netcfg.yaml to add the new name (which was the old name with 'np0' at the end) and enable dhcp for it.

This is a very unfortunate regression as all I did was update my kernel and then my system dropped offline.  It took many hours of debugging to determine what was happening and then had to log in locally to fix it.  There were no messages in dmesg that I could find that gave any clue as to what happened or where to look.

Per https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html, to me this clearly qualifies as a regression because in no way was I prepared for my system to randomly drop offline just because I updated the kernel.

--David Kaplan

Re: [Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Jakub Kicinski]

On Tue, 17 Jun 2025 11:01:18 +0200 Przemek Kitszel wrote:
> > Normally, the device is given a name like enp194s0f0 and connects to the wired network.    
> 
> now the name is likely to be different, please see this thread:
> https://lkml.org/lkml/2025/4/24/1750
> 
> Is it possible that your network setup script has some part of interface
> discovery hardcoded?

Hi Przemek, could you/someone on your side try the workaround 
I suggested in the earlier thread? I'm not sure how actively 
developed ixgbe is, if the work is not related to any new devices 
with more complicated port topology breaking people's setups feels 
pretty unnecessary.

Re: [Intel-wired-lan] [REGRESSION] ixgbe broken after devlink support patch

[Przemek Kitszel]

On 6/18/25 01:09, Jakub Kicinski wrote:
> On Tue, 17 Jun 2025 11:01:18 +0200 Przemek Kitszel wrote:
>>> Normally, the device is given a name like enp194s0f0 and connects to the wired network.
>>
>> now the name is likely to be different, please see this thread:
>> https://lkml.org/lkml/2025/4/24/1750
>>
>> Is it possible that your network setup script has some part of interface
>> discovery hardcoded?
> 
> Hi Przemek, could you/someone on your side try the workaround
> I suggested in the earlier thread? I'm not sure how actively
> developed ixgbe is, if the work is not related to any new devices
> with more complicated port topology breaking people's setups feels
> pretty unnecessary.


yeah, we will fix that, thank you for the remainder