On Fri Jul 14 2023, Heiner Kallweit wrote:
>>> Thanks for the report. It's interesting that the issue seems to occur only on systems
>>> where BIOS doesn't allow OS to control ASPM. Maybe this results in the PCI subsystem
>>> not properly initializing something.
>>> Kurt/Klaus: Could you please boot with cmd line parameter pcie_aspm=force and see
>>> whether this changes something?
>>> This parameter lets Linux ignore the BIOS setting. You should see a message
>>> "PCIe ASPM is forcibly enabled" in the dmesg log with this parameter.
>> 
>> Seems like this does not help. There are still PCIe errors:
>> 
>> |~ # dmesg | grep -i ASPM
>> |[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.4.2-gentoo-kurtOS root=/dev/nvme0n1p3 ro kvm-intel.nested=1 vga=794 pcie_aspm=force
>> |[    0.044016] Kernel command line: BOOT_IMAGE=/vmlinuz-6.4.2-gentoo-kurtOS root=/dev/nvme0n1p3 ro kvm-intel.nested=1 vga=794 pcie_aspm=force
>> |[    0.044048] PCIe ASPM is forcibly enabled
>> |[    0.153011] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
>> |[    0.916341] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
>> |[    0.917719] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
>> |~ # dmesg | grep -i r8169
>> |[    1.337417] r8169 0000:03:00.0 eth0: RTL8168h/8111h, 6c:3c:8c:2c:bd:de, XID 541, IRQ 164
>> |[    1.337422] r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
>> |[    2.833876] r8169 0000:03:00.0 enp3s0: renamed from eth0
>> |[   20.886564] Generic FE-GE Realtek PHY r8169-0-300:00: attached PHY driver (mii_bus:phy_addr=r8169-0-300:00, irq=MAC)
>> |[   21.168373] r8169 0000:03:00.0 enp3s0: Link is Down
>> |[   24.006543] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control off
>> |~ # dmesg | tail
>> |[   20.886564] Generic FE-GE Realtek PHY r8169-0-300:00: attached PHY driver (mii_bus:phy_addr=r8169-0-300:00, irq=MAC)
>> |[   21.168373] r8169 0000:03:00.0 enp3s0: Link is Down
>> |[   24.006543] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control off
>> |[   24.006568] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
>> |[   24.567803] ACPI Warning: \_SB.PC00.PEG1.PEGP._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230331/nsarguments-61)
>> |[   41.563396] pcieport 0000:00:1c.2: AER: Corrected error received: 0000:03:00.0
>> |[   47.065441] pcieport 0000:00:1c.2: AER: Multiple Corrected error received: 0000:03:00.0
>> |[   54.264285] pcieport 0000:00:1c.2: AER: Corrected error received: 0000:03:00.0
>> |[   54.424210] pcieport 0000:00:1c.2: AER: Corrected error received: 0000:03:00.0
>> |[   55.443439] pcieport 0000:00:1c.2: AER: Corrected error received: 0000:03:00.0
>> 
>
> But no tx timeout (yet)?

No, not yet. It doesn't trigger immediately.

> Now that ASPM is forced, could you please disable ASPM L1.2?
> -> /sys/class/net/enp3s0/device/link/l1_2_aspm
> That's what we did until 6.3 for RTL8168h on systems where
> OS can control ASPM.

OK. Disabled ASPM L1.2. PCIe error messages stopped, but the NIC ran
into tx timeout:

|[Fri Jul 14 09:54:00 2023] ------------[ cut here ]------------
|[Fri Jul 14 09:54:00 2023] NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out 9200 ms
|[Fri Jul 14 09:54:00 2023] WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x176/0x1ea
|[...]
|[Fri Jul 14 09:54:03 2023] pcieport 0000:00:1c.2: Data Link Layer Link Active not set in 1000 msec
|[Fri Jul 14 09:54:03 2023] r8169 0000:03:00.0 enp3s0: Can't reset secondary PCI bus, detach NIC

Thanks,
Kurt