[REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2174 bytes --]

Hi,

After updating to 6.14.2, the ethernet adapter is almost unusable, I get
over 30% packet loss.
Bisect says it's this commit:

    commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b
    Author: Vitaly Lifshits <vitaly.lifshits@intel.com>
    Date:   Thu Mar 13 16:05:56 2025 +0200

        e1000e: change k1 configuration on MTP and later platforms
        
        [ Upstream commit efaaf344bc2917cbfa5997633bc18a05d3aed27f ]
        
        Starting from Meteor Lake, the Kumeran interface between the integrated
        MAC and the I219 PHY works at a different frequency. This causes sporadic
        MDI errors when accessing the PHY, and in rare circumstances could lead
        to packet corruption.
        
        To overcome this, introduce minor changes to the Kumeran idle
        state (K1) parameters during device initialization. Hardware reset
        reverts this configuration, therefore it needs to be applied in a few
        places.
        
        Fixes: cc23f4f0b6b9 ("e1000e: Add support for Meteor Lake")
        Signed-off-by: Vitaly Lifshits <vitaly.lifshits@intel.com>
        Tested-by: Avigail Dahan <avigailx.dahan@intel.com>
        Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
        Signed-off-by: Sasha Levin <sashal@kernel.org>

     drivers/net/ethernet/intel/e1000e/defines.h |  3 ++
     drivers/net/ethernet/intel/e1000e/ich8lan.c | 80 +++++++++++++++++++++++++++--
     drivers/net/ethernet/intel/e1000e/ich8lan.h |  4 ++
     3 files changed, 82 insertions(+), 5 deletions(-)

My system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And
the e1000e driver is running in a Xen HVM (with PCI passthrough).
Interestingly, I have also another one with Intel Core Ultra 7 155H
where the issue does not happen. I don't see what is different about
network adapter there, they look identical on lspci (but there are
differences about other devices)...

I see the commit above was already backported to other stable branches
too...

#regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

On 4/14/2025 3:18 PM, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> After updating to 6.14.2, the ethernet adapter is almost unusable, I get
> over 30% packet loss.
> Bisect says it's this commit:
> 
>      commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b
>      Author: Vitaly Lifshits <vitaly.lifshits@intel.com>
>      Date:   Thu Mar 13 16:05:56 2025 +0200
> 
>          e1000e: change k1 configuration on MTP and later platforms
> 
> My system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And
> the e1000e driver is running in a Xen HVM (with PCI passthrough).
> Interestingly, I have also another one with Intel Core Ultra 7 155H
> where the issue does not happen. I don't see what is different about
> network adapter there, they look identical on lspci (but there are
> differences about other devices)...
> 
> I see the commit above was already backported to other stable branches
> too...
> 
> #regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b
> 

Thank you for this report.

Do you see the high packet loss without the virtualization?
Can you please share the lspci output?
Does your switch/link partner support flow control? if it is 
configurable can you try to enable it?
Do you see any errors in dmesg related to the e1000e driver?

Re: [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3603 bytes --]

On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
> Do you see the high packet loss without the virtualization?

I can't check that easily right now, will try later.

> Can you please share the lspci output?

Sure:

00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20)
	Subsystem: CLEVO/KAPOK Computer Device [1558:a743]
	Physical Slot: 7
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Interrupt: pin D routed to IRQ 69
	Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K]
	Capabilities: [c8] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 0000
	Kernel driver in use: e1000e
	Kernel modules: e1000e



> Does your switch/link partner support flow control? if it is configurable
> can you try to enable it?

It does support it. Enabling it makes things much worse...

> Do you see any errors in dmesg related to the e1000e driver?

Not really.
dmesg | grep 'e1000e\|ens7':

[    3.088489] e1000e: Intel(R) PRO/1000 Network Driver
[    3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock
[    3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb
[    3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection
[    3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF
[    3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0
[    8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down
[  242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

(you can also see a test with flow control above)


And also ethtool output if useful:
Settings for ens7:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Full
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	MDI-X: on (auto)
	Supports Wake-on: d
	Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

On 4/14/2025 3:58 PM, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
>> Do you see the high packet loss without the virtualization?
> 
> I can't check that easily right now, will try later.
> 
>> Can you please share the lspci output?
> 
> Sure:
> 
> 00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20)
> 	Subsystem: CLEVO/KAPOK Computer Device [1558:a743]
> 	Physical Slot: 7
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 64
> 	Interrupt: pin D routed to IRQ 69
> 	Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K]
> 	Capabilities: [c8] Power Management version 3
> 		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> 		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> 	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
> 		Address: 00000000fee00000  Data: 0000
> 	Kernel driver in use: e1000e
> 	Kernel modules: e1000e
> 


Do you have mei modules running? Can you try if disabling them make 
things better?

> 
> 
>> Does your switch/link partner support flow control? if it is configurable
>> can you try to enable it?
> 
> It does support it. Enabling it makes things much worse...
> 
>> Do you see any errors in dmesg related to the e1000e driver?
> 
> Not really.
> dmesg | grep 'e1000e\|ens7':
> 
> [    3.088489] e1000e: Intel(R) PRO/1000 Network Driver
> [    3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
> [    3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> [    3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock
> [    3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb
> [    3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection
> [    3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF
> [    3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0
> [    8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
> [  239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down
> [  242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
> 
> (you can also see a test with flow control above)
> 
> 
> And also ethtool output if useful:
> Settings for ens7:
> 	Supported ports: [ TP ]
> 	Supported link modes:   10baseT/Half 10baseT/Full
> 	                        100baseT/Half 100baseT/Full
> 	                        1000baseT/Full
> 	Supported pause frame use: Symmetric Receive-only
> 	Supports auto-negotiation: Yes
> 	Supported FEC modes: Not reported
> 	Advertised link modes:  10baseT/Half 10baseT/Full
> 	                        100baseT/Half 100baseT/Full
> 	                        1000baseT/Full
> 	Advertised pause frame use: Symmetric Receive-only
> 	Advertised auto-negotiation: Yes
> 	Advertised FEC modes: Not reported
> 	Link partner advertised link modes:  10baseT/Half 10baseT/Full
> 	                                     100baseT/Half 100baseT/Full
> 	                                     1000baseT/Full
> 	Link partner advertised pause frame use: No
> 	Link partner advertised auto-negotiation: Yes
> 	Link partner advertised FEC modes: Not reported
> 	Speed: 1000Mb/s
> 	Duplex: Full
> 	Auto-negotiation: on
> 	Port: Twisted Pair
> 	PHYAD: 1
> 	Transceiver: internal
> 	MDI-X: on (auto)
> 	Supports Wake-on: d
> 	Wake-on: d
>          Current message level: 0x00000007 (7)
>                                 drv probe link
> 	Link detected: yes
> 
> 

Re: [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On Mon, Apr 14, 2025 at 04:04:51PM +0300, Lifshits, Vitaly wrote:
> Do you have mei modules running? Can you try if disabling them make things
> better?

I do, disabling them (via module_blacklist=mei) doesn't help.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

>> And also ethtool output if useful:
>> Settings for ens7:
>>     Supported ports: [ TP ]
>>     Supported link modes:   10baseT/Half 10baseT/Full
>>                             100baseT/Half 100baseT/Full
>>                             1000baseT/Full
>>     Supported pause frame use: Symmetric Receive-only
>>     Supports auto-negotiation: Yes
>>     Supported FEC modes: Not reported
>>     Advertised link modes:  10baseT/Half 10baseT/Full
>>                             100baseT/Half 100baseT/Full
>>                             1000baseT/Full
>>     Advertised pause frame use: Symmetric Receive-only
>>     Advertised auto-negotiation: Yes
>>     Advertised FEC modes: Not reported
>>     Link partner advertised link modes:  10baseT/Half 10baseT/Full
>>                                          100baseT/Half 100baseT/Full
>>                                          1000baseT/Full
>>     Link partner advertised pause frame use: No
>>     Link partner advertised auto-negotiation: Yes
>>     Link partner advertised FEC modes: Not reported
>>     Speed: 1000Mb/s
>>     Duplex: Full
>>     Auto-negotiation: on
>>     Port: Twisted Pair
>>     PHYAD: 1
>>     Transceiver: internal
>>     MDI-X: on (auto)
>>     Supports Wake-on: d
>>     Wake-on: d
>>          Current message level: 0x00000007 (7)
>>                                 drv probe link
>>     Link detected: yes
>>
>>

Can you please also share the output of ethtool -i? I would like to know 
the NVM version that you have on your device.

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> Can you please also share the output of ethtool -i? I would like to know the
> NVM version that you have on your device.

driver: e1000e
version: 6.14.1+
firmware-version: 1.1-4
expansion-rom-version: 
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
>> Can you please also share the output of ethtool -i? I would like to know the
>> NVM version that you have on your device.
> 
> driver: e1000e
> version: 6.14.1+
> firmware-version: 1.1-4
> expansion-rom-version:
> bus-info: 0000:00:1f.6
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> 

Your firmware version is not the latest, can you check with the board 
manufacturer if there is a BIOS update to your system?

Also, you mentioned that on another system this issue doesn't reproduce, 
do they have the same firmware version?

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> 
> 
> On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > Can you please also share the output of ethtool -i? I would like to know the
> > > NVM version that you have on your device.
> > 
> > driver: e1000e
> > version: 6.14.1+
> > firmware-version: 1.1-4
> > expansion-rom-version:
> > bus-info: 0000:00:1f.6
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: yes
> > 
> 
> Your firmware version is not the latest, can you check with the board
> manufacturer if there is a BIOS update to your system?

I can check, but still, it's a regression in the Linux driver - old
kernel did work perfectly well on this hw. Maybe new driver tries to use
some feature that is missing (or broken) in the old firmware?

> Also, you mentioned that on another system this issue doesn't reproduce, do
> they have the same firmware version?

The other one has also 1.1-4 firmware. And I re-checked, e1000e from
6.14.2 works fine there.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1792 bytes --]

On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > 
> > 
> > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > NVM version that you have on your device.
> > > 
> > > driver: e1000e
> > > version: 6.14.1+
> > > firmware-version: 1.1-4
> > > expansion-rom-version:
> > > bus-info: 0000:00:1f.6
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: yes
> > > 
> > 
> > Your firmware version is not the latest, can you check with the board
> > manufacturer if there is a BIOS update to your system?
> 
> I can check, but still, it's a regression in the Linux driver - old
> kernel did work perfectly well on this hw. Maybe new driver tries to use
> some feature that is missing (or broken) in the old firmware?

A little bit of context: I'm maintaining the kernel package for a Qubes
OS distribution. While I can try to update firmware on my test system, I
have no influence on what hardware users will use this kernel, and
which firmware version they will use (and whether all the vendors
provide newer firmware at all). I cannot ship a kernel that is known
to break network on some devices.

> > Also, you mentioned that on another system this issue doesn't reproduce, do
> > they have the same firmware version?
> 
> The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> 6.14.2 works fine there.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
>> On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
>>>
>>>
>>> On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
>>>> On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
>>>>> Can you please also share the output of ethtool -i? I would like to know the
>>>>> NVM version that you have on your device.
>>>>
>>>> driver: e1000e
>>>> version: 6.14.1+
>>>> firmware-version: 1.1-4
>>>> expansion-rom-version:
>>>> bus-info: 0000:00:1f.6
>>>> supports-statistics: yes
>>>> supports-test: yes
>>>> supports-eeprom-access: yes
>>>> supports-register-dump: yes
>>>> supports-priv-flags: yes
>>>>
>>>
>>> Your firmware version is not the latest, can you check with the board
>>> manufacturer if there is a BIOS update to your system?
>>
>> I can check, but still, it's a regression in the Linux driver - old
>> kernel did work perfectly well on this hw. Maybe new driver tries to use
>> some feature that is missing (or broken) in the old firmware?
> 
> A little bit of context: I'm maintaining the kernel package for a Qubes
> OS distribution. While I can try to update firmware on my test system, I
> have no influence on what hardware users will use this kernel, and
> which firmware version they will use (and whether all the vendors
> provide newer firmware at all). I cannot ship a kernel that is known
> to break network on some devices.
> 
>>> Also, you mentioned that on another system this issue doesn't reproduce, do
>>> they have the same firmware version?
>>
>> The other one has also 1.1-4 firmware. And I re-checked, e1000e from
>> 6.14.2 works fine there.
> 

Dear Marek,

Thank you for your detailed feedback and for providing the requested 
information.

We have conducted extensive testing of this patch across multiple 
systems and have not observed any packet loss issues. Upon comparing the 
mentioned setups, we noted that while the LAN controller is similar, the 
CPU differs. We believe that the issue may be related to transitions in 
the CPU's low power states.

Consequently, we kindly request that you disable the CPU low power state 
transitions in the S0 system state and verify if the issue persists. You 
can disable this in the kernel parameters on the command line with 
idle=poll. Please note that this command is intended for debugging 
purposes only, as it may result in higher power consumption.

Please inform us if disabling the CPU low power states resolves the 
issue or if further investigation is required. As previously mentioned, 
this patch is critical for the operation of Meteor Lake LAN devices, and 
therefore, we are unable to revert it.

Thank you for your cooperation.

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3463 bytes --]

On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly wrote:
> 
> 
> On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > 
> > > > 
> > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > NVM version that you have on your device.
> > > > > 
> > > > > driver: e1000e
> > > > > version: 6.14.1+
> > > > > firmware-version: 1.1-4
> > > > > expansion-rom-version:
> > > > > bus-info: 0000:00:1f.6
> > > > > supports-statistics: yes
> > > > > supports-test: yes
> > > > > supports-eeprom-access: yes
> > > > > supports-register-dump: yes
> > > > > supports-priv-flags: yes
> > > > > 
> > > > 
> > > > Your firmware version is not the latest, can you check with the board
> > > > manufacturer if there is a BIOS update to your system?
> > > 
> > > I can check, but still, it's a regression in the Linux driver - old
> > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > some feature that is missing (or broken) in the old firmware?
> > 
> > A little bit of context: I'm maintaining the kernel package for a Qubes
> > OS distribution. While I can try to update firmware on my test system, I
> > have no influence on what hardware users will use this kernel, and
> > which firmware version they will use (and whether all the vendors
> > provide newer firmware at all). I cannot ship a kernel that is known
> > to break network on some devices.
> > 
> > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > they have the same firmware version?
> > > 
> > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > 6.14.2 works fine there.
> > 
> 
> Dear Marek,
> 
> Thank you for your detailed feedback and for providing the requested
> information.
> 
> We have conducted extensive testing of this patch across multiple systems
> and have not observed any packet loss issues. Upon comparing the mentioned
> setups, we noted that while the LAN controller is similar, the CPU differs.
> We believe that the issue may be related to transitions in the CPU's low
> power states.
> 
> Consequently, we kindly request that you disable the CPU low power state
> transitions in the S0 system state and verify if the issue persists. You can
> disable this in the kernel parameters on the command line with idle=poll.
> Please note that this command is intended for debugging purposes only, as it
> may result in higher power consumption.

I tried with idle=poll, and it didn't help, I still see a lot of packet
losses. But I can also confirm that idle=poll makes the system use
significantly more power (previously at 25-30W, with this option stays
at about 42W).

Is there any other info I can provide, enable some debug features or
something?

I see the problem is with receiving packets - in my simple ping test,
the ping target sees all the echo requests (and respond to them), but
the responses aren't reaching ping back (and are not visible on tcpdump
on the problematic system either).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Paul Menzel]

Dear Marek, dear Vitaly,


Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly 
>> On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
>>> On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
>>>> On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
>>>>>
>>>>>
>>>>> On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
>>>>>> On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
>>>>>>> Can you please also share the output of ethtool -i? I would like to know the
>>>>>>> NVM version that you have on your device.
>>>>>>
>>>>>> driver: e1000e
>>>>>> version: 6.14.1+
>>>>>> firmware-version: 1.1-4
>>>>>> expansion-rom-version:
>>>>>> bus-info: 0000:00:1f.6
>>>>>> supports-statistics: yes
>>>>>> supports-test: yes
>>>>>> supports-eeprom-access: yes
>>>>>> supports-register-dump: yes
>>>>>> supports-priv-flags: yes
>>>>>>
>>>>>
>>>>> Your firmware version is not the latest, can you check with the board
>>>>> manufacturer if there is a BIOS update to your system?
>>>>
>>>> I can check, but still, it's a regression in the Linux driver - old
>>>> kernel did work perfectly well on this hw. Maybe new driver tries to use
>>>> some feature that is missing (or broken) in the old firmware?
>>>
>>> A little bit of context: I'm maintaining the kernel package for a Qubes
>>> OS distribution. While I can try to update firmware on my test system, I
>>> have no influence on what hardware users will use this kernel, and
>>> which firmware version they will use (and whether all the vendors
>>> provide newer firmware at all). I cannot ship a kernel that is known
>>> to break network on some devices.
>>>
>>>>> Also, you mentioned that on another system this issue doesn't reproduce, do
>>>>> they have the same firmware version?
>>>>
>>>> The other one has also 1.1-4 firmware. And I re-checked, e1000e from
>>>> 6.14.2 works fine there.

>> Thank you for your detailed feedback and for providing the requested
>> information.
>>
>> We have conducted extensive testing of this patch across multiple systems
>> and have not observed any packet loss issues. Upon comparing the mentioned
>> setups, we noted that while the LAN controller is similar, the CPU differs.
>> We believe that the issue may be related to transitions in the CPU's low
>> power states.
>>
>> Consequently, we kindly request that you disable the CPU low power state
>> transitions in the S0 system state and verify if the issue persists. You can
>> disable this in the kernel parameters on the command line with idle=poll.
>> Please note that this command is intended for debugging purposes only, as it
>> may result in higher power consumption.
> 
> I tried with idle=poll, and it didn't help, I still see a lot of packet
> losses. But I can also confirm that idle=poll makes the system use
> significantly more power (previously at 25-30W, with this option stays
> at about 42W).
> 
> Is there any other info I can provide, enable some debug features or
> something?
> 
> I see the problem is with receiving packets - in my simple ping test,
> the ping target sees all the echo requests (and respond to them), but
> the responses aren't reaching ping back (and are not visible on tcpdump
> on the problematic system either).

As the cause is still unclear, can the commit please be reverted in the 
master branch due adhere to Linux’ no-regression policy, so that it can 
be reverted from the stable series?

Marek, did you also test 6.15 release candidates?


Kind regards,

Paul

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]

On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> Dear Marek, dear Vitaly,
> 
> 
> Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > 
> > > > > > 
> > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > NVM version that you have on your device.
> > > > > > > 
> > > > > > > driver: e1000e
> > > > > > > version: 6.14.1+
> > > > > > > firmware-version: 1.1-4
> > > > > > > expansion-rom-version:
> > > > > > > bus-info: 0000:00:1f.6
> > > > > > > supports-statistics: yes
> > > > > > > supports-test: yes
> > > > > > > supports-eeprom-access: yes
> > > > > > > supports-register-dump: yes
> > > > > > > supports-priv-flags: yes
> > > > > > > 
> > > > > > 
> > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > manufacturer if there is a BIOS update to your system?
> > > > > 
> > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > some feature that is missing (or broken) in the old firmware?
> > > > 
> > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > OS distribution. While I can try to update firmware on my test system, I
> > > > have no influence on what hardware users will use this kernel, and
> > > > which firmware version they will use (and whether all the vendors
> > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > to break network on some devices.
> > > > 
> > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > they have the same firmware version?
> > > > > 
> > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > 6.14.2 works fine there.
> 
> > > Thank you for your detailed feedback and for providing the requested
> > > information.
> > > 
> > > We have conducted extensive testing of this patch across multiple systems
> > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > We believe that the issue may be related to transitions in the CPU's low
> > > power states.
> > > 
> > > Consequently, we kindly request that you disable the CPU low power state
> > > transitions in the S0 system state and verify if the issue persists. You can
> > > disable this in the kernel parameters on the command line with idle=poll.
> > > Please note that this command is intended for debugging purposes only, as it
> > > may result in higher power consumption.
> > 
> > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > losses. But I can also confirm that idle=poll makes the system use
> > significantly more power (previously at 25-30W, with this option stays
> > at about 42W).
> > 
> > Is there any other info I can provide, enable some debug features or
> > something?
> > 
> > I see the problem is with receiving packets - in my simple ping test,
> > the ping target sees all the echo requests (and respond to them), but
> > the responses aren't reaching ping back (and are not visible on tcpdump
> > on the problematic system either).
> 
> As the cause is still unclear, can the commit please be reverted in the
> master branch due adhere to Linux’ no-regression policy, so that it can be
> reverted from the stable series?
> 
> Marek, did you also test 6.15 release candidates?

The last test I did was on 6.15-rc3. I can re-test on -rc5.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4440 bytes --]

On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> > Dear Marek, dear Vitaly,
> > 
> > 
> > Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > > NVM version that you have on your device.
> > > > > > > > 
> > > > > > > > driver: e1000e
> > > > > > > > version: 6.14.1+
> > > > > > > > firmware-version: 1.1-4
> > > > > > > > expansion-rom-version:
> > > > > > > > bus-info: 0000:00:1f.6
> > > > > > > > supports-statistics: yes
> > > > > > > > supports-test: yes
> > > > > > > > supports-eeprom-access: yes
> > > > > > > > supports-register-dump: yes
> > > > > > > > supports-priv-flags: yes
> > > > > > > > 
> > > > > > > 
> > > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > > manufacturer if there is a BIOS update to your system?
> > > > > > 
> > > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > > some feature that is missing (or broken) in the old firmware?
> > > > > 
> > > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > > OS distribution. While I can try to update firmware on my test system, I
> > > > > have no influence on what hardware users will use this kernel, and
> > > > > which firmware version they will use (and whether all the vendors
> > > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > > to break network on some devices.
> > > > > 
> > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > > they have the same firmware version?
> > > > > > 
> > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > > 6.14.2 works fine there.
> > 
> > > > Thank you for your detailed feedback and for providing the requested
> > > > information.
> > > > 
> > > > We have conducted extensive testing of this patch across multiple systems
> > > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > > We believe that the issue may be related to transitions in the CPU's low
> > > > power states.
> > > > 
> > > > Consequently, we kindly request that you disable the CPU low power state
> > > > transitions in the S0 system state and verify if the issue persists. You can
> > > > disable this in the kernel parameters on the command line with idle=poll.
> > > > Please note that this command is intended for debugging purposes only, as it
> > > > may result in higher power consumption.
> > > 
> > > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > > losses. But I can also confirm that idle=poll makes the system use
> > > significantly more power (previously at 25-30W, with this option stays
> > > at about 42W).
> > > 
> > > Is there any other info I can provide, enable some debug features or
> > > something?
> > > 
> > > I see the problem is with receiving packets - in my simple ping test,
> > > the ping target sees all the echo requests (and respond to them), but
> > > the responses aren't reaching ping back (and are not visible on tcpdump
> > > on the problematic system either).
> > 
> > As the cause is still unclear, can the commit please be reverted in the
> > master branch due adhere to Linux’ no-regression policy, so that it can be
> > reverted from the stable series?
> > 
> > Marek, did you also test 6.15 release candidates?
> 
> The last test I did was on 6.15-rc3. I can re-test on -rc5.

Same with 6.15-rc5.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4898 bytes --]

On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
> > On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> > > Dear Marek, dear Vitaly,
> > > 
> > > 
> > > Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > > > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > > > NVM version that you have on your device.
> > > > > > > > > 
> > > > > > > > > driver: e1000e
> > > > > > > > > version: 6.14.1+
> > > > > > > > > firmware-version: 1.1-4
> > > > > > > > > expansion-rom-version:
> > > > > > > > > bus-info: 0000:00:1f.6
> > > > > > > > > supports-statistics: yes
> > > > > > > > > supports-test: yes
> > > > > > > > > supports-eeprom-access: yes
> > > > > > > > > supports-register-dump: yes
> > > > > > > > > supports-priv-flags: yes
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > > > manufacturer if there is a BIOS update to your system?
> > > > > > > 
> > > > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > > > some feature that is missing (or broken) in the old firmware?
> > > > > > 
> > > > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > > > OS distribution. While I can try to update firmware on my test system, I
> > > > > > have no influence on what hardware users will use this kernel, and
> > > > > > which firmware version they will use (and whether all the vendors
> > > > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > > > to break network on some devices.
> > > > > > 
> > > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > > > they have the same firmware version?
> > > > > > > 
> > > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > > > 6.14.2 works fine there.
> > > 
> > > > > Thank you for your detailed feedback and for providing the requested
> > > > > information.
> > > > > 
> > > > > We have conducted extensive testing of this patch across multiple systems
> > > > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > > > We believe that the issue may be related to transitions in the CPU's low
> > > > > power states.
> > > > > 
> > > > > Consequently, we kindly request that you disable the CPU low power state
> > > > > transitions in the S0 system state and verify if the issue persists. You can
> > > > > disable this in the kernel parameters on the command line with idle=poll.
> > > > > Please note that this command is intended for debugging purposes only, as it
> > > > > may result in higher power consumption.
> > > > 
> > > > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > > > losses. But I can also confirm that idle=poll makes the system use
> > > > significantly more power (previously at 25-30W, with this option stays
> > > > at about 42W).
> > > > 
> > > > Is there any other info I can provide, enable some debug features or
> > > > something?
> > > > 
> > > > I see the problem is with receiving packets - in my simple ping test,
> > > > the ping target sees all the echo requests (and respond to them), but
> > > > the responses aren't reaching ping back (and are not visible on tcpdump
> > > > on the problematic system either).
> > > 
> > > As the cause is still unclear, can the commit please be reverted in the
> > > master branch due adhere to Linux’ no-regression policy, so that it can be
> > > reverted from the stable series?
> > > 
> > > Marek, did you also test 6.15 release candidates?
> > 
> > The last test I did was on 6.15-rc3. I can re-test on -rc5.
> 
> Same with 6.15-rc5.

And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has
this buggy patch revered and nobody complained (contrary to the version
with the patch included). Should I submit the revert patch?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 5217 bytes --]

On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
> On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
> > On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
> > > On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> > > > Dear Marek, dear Vitaly,
> > > > 
> > > > 
> > > > Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > > > > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > > > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > > > > NVM version that you have on your device.
> > > > > > > > > > 
> > > > > > > > > > driver: e1000e
> > > > > > > > > > version: 6.14.1+
> > > > > > > > > > firmware-version: 1.1-4
> > > > > > > > > > expansion-rom-version:
> > > > > > > > > > bus-info: 0000:00:1f.6
> > > > > > > > > > supports-statistics: yes
> > > > > > > > > > supports-test: yes
> > > > > > > > > > supports-eeprom-access: yes
> > > > > > > > > > supports-register-dump: yes
> > > > > > > > > > supports-priv-flags: yes
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > > > > manufacturer if there is a BIOS update to your system?
> > > > > > > > 
> > > > > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > > > > some feature that is missing (or broken) in the old firmware?
> > > > > > > 
> > > > > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > > > > OS distribution. While I can try to update firmware on my test system, I
> > > > > > > have no influence on what hardware users will use this kernel, and
> > > > > > > which firmware version they will use (and whether all the vendors
> > > > > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > > > > to break network on some devices.
> > > > > > > 
> > > > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > > > > they have the same firmware version?
> > > > > > > > 
> > > > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > > > > 6.14.2 works fine there.
> > > > 
> > > > > > Thank you for your detailed feedback and for providing the requested
> > > > > > information.
> > > > > > 
> > > > > > We have conducted extensive testing of this patch across multiple systems
> > > > > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > > > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > > > > We believe that the issue may be related to transitions in the CPU's low
> > > > > > power states.
> > > > > > 
> > > > > > Consequently, we kindly request that you disable the CPU low power state
> > > > > > transitions in the S0 system state and verify if the issue persists. You can
> > > > > > disable this in the kernel parameters on the command line with idle=poll.
> > > > > > Please note that this command is intended for debugging purposes only, as it
> > > > > > may result in higher power consumption.
> > > > > 
> > > > > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > > > > losses. But I can also confirm that idle=poll makes the system use
> > > > > significantly more power (previously at 25-30W, with this option stays
> > > > > at about 42W).
> > > > > 
> > > > > Is there any other info I can provide, enable some debug features or
> > > > > something?
> > > > > 
> > > > > I see the problem is with receiving packets - in my simple ping test,
> > > > > the ping target sees all the echo requests (and respond to them), but
> > > > > the responses aren't reaching ping back (and are not visible on tcpdump
> > > > > on the problematic system either).
> > > > 
> > > > As the cause is still unclear, can the commit please be reverted in the
> > > > master branch due adhere to Linux’ no-regression policy, so that it can be
> > > > reverted from the stable series?
> > > > 
> > > > Marek, did you also test 6.15 release candidates?
> > > 
> > > The last test I did was on 6.15-rc3. I can re-test on -rc5.
> > 
> > Same with 6.15-rc5.
> 
> And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has
> this buggy patch revered and nobody complained (contrary to the version
> with the patch included). Should I submit the revert patch?

Just submit a revert then 👍 I have no authority here, but had good
experience with just sending a revert patch in the past 🤗

Cheers,
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Lifshits, Vitaly]

On 6/18/2025 4:41 PM, Christian Heusel wrote:
> On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
>> On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
>>> On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
>>>> On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
>>>>> Dear Marek, dear Vitaly,
>>>>>
>>>>>
>>>>> Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
>>>>>> On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
>>>>>>> On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
>>>>>>>> On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
>>>>>>>>> On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
>>>>>>>>>>> On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
>>>>>>>>>>>> Can you please also share the output of ethtool -i? I would like to know the
>>>>>>>>>>>> NVM version that you have on your device.
>>>>>>>>>>>
>>>>>>>>>>> driver: e1000e
>>>>>>>>>>> version: 6.14.1+
>>>>>>>>>>> firmware-version: 1.1-4
>>>>>>>>>>> expansion-rom-version:
>>>>>>>>>>> bus-info: 0000:00:1f.6
>>>>>>>>>>> supports-statistics: yes
>>>>>>>>>>> supports-test: yes
>>>>>>>>>>> supports-eeprom-access: yes
>>>>>>>>>>> supports-register-dump: yes
>>>>>>>>>>> supports-priv-flags: yes
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Your firmware version is not the latest, can you check with the board
>>>>>>>>>> manufacturer if there is a BIOS update to your system?
>>>>>>>>>
>>>>>>>>> I can check, but still, it's a regression in the Linux driver - old
>>>>>>>>> kernel did work perfectly well on this hw. Maybe new driver tries to use
>>>>>>>>> some feature that is missing (or broken) in the old firmware?
>>>>>>>>
>>>>>>>> A little bit of context: I'm maintaining the kernel package for a Qubes
>>>>>>>> OS distribution. While I can try to update firmware on my test system, I
>>>>>>>> have no influence on what hardware users will use this kernel, and
>>>>>>>> which firmware version they will use (and whether all the vendors
>>>>>>>> provide newer firmware at all). I cannot ship a kernel that is known
>>>>>>>> to break network on some devices.
>>>>>>>>
>>>>>>>>>> Also, you mentioned that on another system this issue doesn't reproduce, do
>>>>>>>>>> they have the same firmware version?
>>>>>>>>>
>>>>>>>>> The other one has also 1.1-4 firmware. And I re-checked, e1000e from
>>>>>>>>> 6.14.2 works fine there.
>>>>>
>>>>>>> Thank you for your detailed feedback and for providing the requested
>>>>>>> information.
>>>>>>>
>>>>>>> We have conducted extensive testing of this patch across multiple systems
>>>>>>> and have not observed any packet loss issues. Upon comparing the mentioned
>>>>>>> setups, we noted that while the LAN controller is similar, the CPU differs.
>>>>>>> We believe that the issue may be related to transitions in the CPU's low
>>>>>>> power states.
>>>>>>>
>>>>>>> Consequently, we kindly request that you disable the CPU low power state
>>>>>>> transitions in the S0 system state and verify if the issue persists. You can
>>>>>>> disable this in the kernel parameters on the command line with idle=poll.
>>>>>>> Please note that this command is intended for debugging purposes only, as it
>>>>>>> may result in higher power consumption.
>>>>>>
>>>>>> I tried with idle=poll, and it didn't help, I still see a lot of packet
>>>>>> losses. But I can also confirm that idle=poll makes the system use
>>>>>> significantly more power (previously at 25-30W, with this option stays
>>>>>> at about 42W).
>>>>>>
>>>>>> Is there any other info I can provide, enable some debug features or
>>>>>> something?
>>>>>>
>>>>>> I see the problem is with receiving packets - in my simple ping test,
>>>>>> the ping target sees all the echo requests (and respond to them), but
>>>>>> the responses aren't reaching ping back (and are not visible on tcpdump
>>>>>> on the problematic system either).
>>>>>
>>>>> As the cause is still unclear, can the commit please be reverted in the
>>>>> master branch due adhere to Linux’ no-regression policy, so that it can be
>>>>> reverted from the stable series?
>>>>>
>>>>> Marek, did you also test 6.15 release candidates?
>>>>
>>>> The last test I did was on 6.15-rc3. I can re-test on -rc5.
>>>
>>> Same with 6.15-rc5.
>>
>> And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has
>> this buggy patch revered and nobody complained (contrary to the version
>> with the patch included). Should I submit the revert patch?

It is not a good idea to revert this patch as most of the systems will 
encounter the original issues (PHY access and packet loss). The reason I 
first introduced this patch was because big vendors reported the packet 
loss issue. You can refer to the following sightings:
https://answers.launchpad.net/ubuntu/+question/816003
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2066064
https://bugzilla.kernel.org/show_bug.cgi?id=218869

As an intermediate solution we can either use a privileged flag to make 
it configurable. I will share with you a patch that might fix the issue
on your system that I would like you to try.

FYI, we are currently investigating a similar issue that seems to be due 
to a misconfiguration of the system firmware.

> 
> Just submit a revert then 👍 I have no authority here, but had good
> experience with just sending a revert patch in the past 🤗
> 
> Cheers,
> Chris

Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 6656 bytes --]

On Thu, Jun 19, 2025 at 03:20:35PM +0300, Lifshits, Vitaly wrote:
> 
> 
> On 6/18/2025 4:41 PM, Christian Heusel wrote:
> > On 25/06/18 03:28PM, Marek Marczykowski-Górecki wrote:
> > > On Fri, May 09, 2025 at 02:17:32AM +0200, Marek Marczykowski-Górecki wrote:
> > > > On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
> > > > > On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> > > > > > Dear Marek, dear Vitaly,
> > > > > > 
> > > > > > 
> > > > > > Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > > > > > > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > > > > > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > > > > > > NVM version that you have on your device.
> > > > > > > > > > > > 
> > > > > > > > > > > > driver: e1000e
> > > > > > > > > > > > version: 6.14.1+
> > > > > > > > > > > > firmware-version: 1.1-4
> > > > > > > > > > > > expansion-rom-version:
> > > > > > > > > > > > bus-info: 0000:00:1f.6
> > > > > > > > > > > > supports-statistics: yes
> > > > > > > > > > > > supports-test: yes
> > > > > > > > > > > > supports-eeprom-access: yes
> > > > > > > > > > > > supports-register-dump: yes
> > > > > > > > > > > > supports-priv-flags: yes
> > > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > > > > > > manufacturer if there is a BIOS update to your system?
> > > > > > > > > > 
> > > > > > > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > > > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > > > > > > some feature that is missing (or broken) in the old firmware?
> > > > > > > > > 
> > > > > > > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > > > > > > OS distribution. While I can try to update firmware on my test system, I
> > > > > > > > > have no influence on what hardware users will use this kernel, and
> > > > > > > > > which firmware version they will use (and whether all the vendors
> > > > > > > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > > > > > > to break network on some devices.
> > > > > > > > > 
> > > > > > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > > > > > > they have the same firmware version?
> > > > > > > > > > 
> > > > > > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > > > > > > 6.14.2 works fine there.
> > > > > > 
> > > > > > > > Thank you for your detailed feedback and for providing the requested
> > > > > > > > information.
> > > > > > > > 
> > > > > > > > We have conducted extensive testing of this patch across multiple systems
> > > > > > > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > > > > > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > > > > > > We believe that the issue may be related to transitions in the CPU's low
> > > > > > > > power states.
> > > > > > > > 
> > > > > > > > Consequently, we kindly request that you disable the CPU low power state
> > > > > > > > transitions in the S0 system state and verify if the issue persists. You can
> > > > > > > > disable this in the kernel parameters on the command line with idle=poll.
> > > > > > > > Please note that this command is intended for debugging purposes only, as it
> > > > > > > > may result in higher power consumption.
> > > > > > > 
> > > > > > > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > > > > > > losses. But I can also confirm that idle=poll makes the system use
> > > > > > > significantly more power (previously at 25-30W, with this option stays
> > > > > > > at about 42W).
> > > > > > > 
> > > > > > > Is there any other info I can provide, enable some debug features or
> > > > > > > something?
> > > > > > > 
> > > > > > > I see the problem is with receiving packets - in my simple ping test,
> > > > > > > the ping target sees all the echo requests (and respond to them), but
> > > > > > > the responses aren't reaching ping back (and are not visible on tcpdump
> > > > > > > on the problematic system either).
> > > > > > 
> > > > > > As the cause is still unclear, can the commit please be reverted in the
> > > > > > master branch due adhere to Linux’ no-regression policy, so that it can be
> > > > > > reverted from the stable series?
> > > > > > 
> > > > > > Marek, did you also test 6.15 release candidates?
> > > > > 
> > > > > The last test I did was on 6.15-rc3. I can re-test on -rc5.
> > > > 
> > > > Same with 6.15-rc5.
> > > 
> > > And the same issue still applies to 6.16-rc2. FWIW Qubes OS kernel has
> > > this buggy patch revered and nobody complained (contrary to the version
> > > with the patch included). Should I submit the revert patch?
> 
> It is not a good idea to revert this patch as most of the systems will
> encounter the original issues (PHY access and packet loss). The reason I
> first introduced this patch was because big vendors reported the packet loss
> issue. You can refer to the following sightings:
> https://answers.launchpad.net/ubuntu/+question/816003
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2066064
> https://bugzilla.kernel.org/show_bug.cgi?id=218869

It would be useful to have any of those links in the original commit...

> As an intermediate solution we can either use a privileged flag to make it
> configurable. I will share with you a patch that might fix the issue
> on your system that I would like you to try.

Yes, that patch works :)

> FYI, we are currently investigating a similar issue that seems to be due to
> a misconfiguration of the system firmware.

Can you share some details? I can forward the info to firmware
developers for this system (it's Dasharo - coreboot-based firmware).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [REGRESSION] e1000e heavy packet loss on Meteor Lake - 6.14.2

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

On Mon, Apr 14, 2025 at 02:58:09PM +0200, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
> > Do you see the high packet loss without the virtualization?
> 
> I can't check that easily right now, will try later.

Tried now, the same issue is without any virtualization too.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]