Re: [PATCH v2] PCI: dw-rockchip: Enable async probe by default

[Grimmauld <grimmauld@grimmauld.de>]

Hello Anand,

I have tested this patch.
Hardware/Kernel information:
- radxa rock 5c lite
- rk3588s CPU, arm64
- defconfig NixOS kernel
- picked onto 6.18.7
- DT: rockchip/rk3588s-rock-5c.dtb
- tested both uboot (mainline) and edk2 (vendor)

On Fri, Aug 09, 2024 at 01:06:09PM +0530, Anand Moon wrote:
> Rockchip DWC PCIe driver currently waits for the combo PHY link
> (PCIe 3.0, PCIe 2.0, and SATA 3.0) to be established link training
> during boot, it also waits for the link to be up, which could consume
> several milliseconds during boot.

I found that without this patch, USB 3 ports as well as the PCIe connector seemingly stay uninitialized during boot on my hardware. 
This manifests in a bootable USB flash drive loading initrd from bootloader (both uboot and edk2) perfectly, but then fails to mount the rootfs from the drive.
In effect, boot is not just slower than it should be, it just does not boot all the way at all.
In that scenario, the devfs entries corresponding to the flash drive are simply missing, same for the sysfs where i would expect the USB device listed.
Replugging the USB flash drive during initrd does seem to fix that, but is tedious and not viable for a server.
Similarly, booting from m.2 SSD attached via PCIe fails the same way, with rootfs timing out despite the bootloader correctly reading initrd on the same drive.
Fwiw, replugging the SSD does not work like it does for USB flash drives, and is even worse of an idea.
USB 2 ports as well as SD card boots correctly, even without your patch.
Without your patch, i am seeing "deferred probe pending" in dmesg before boot gets stuck, which was the hint which made me find your patch.
I am not sure whether that is the actual cause or just a symptom for why drives are not recognized during boot, and am not quite sure how to debug this further.

> To optimize boot time, this commit allows asynchronous probing.
> This change enables the PCIe link establishment to occur in the
> background while other devices are being probed.

With this patch, booting from SSD or USB 3 port works flawlessly, and i have not seen any regressions with SD card or USB 2 boot, nor any other hardware component.
This setup has worked for multiple boots without fail, both with traditional ext4 and zfs rootfs being loaded from USB 3 and PCIe.
Because i require this patch to run my rock 5c from SSD, i am currently running a custom patched kernel, but would highly appreciate this patch making its way to mainline eventually.
There might well be something else going on here. The proposed patch may not be the "proper" fix to the issues i am seeing, but it does at least work.

I have NOT tested boot from eMMC (either with or without this patch), though i have no reason to believe it would be impacted.

I am happy to provide more info as needed. First time posting to the LKML, i hope i am doing this right...

Tested-by: Grimmauld <grimmauld@grimmauld.de>

Regards,
Grimmauld