Re: [Regression] PowerNV: JMicron JMB58x AHCI 32-bit DMA fallback triggers EEH freeze on 7.x kernels

[Ritesh Harjani (IBM) <ritesh.list@gmail.com>]

Hi Paul,

Paul Klapperich <paul.klapperich@gmail.com> writes:

> Hi everyone,
>
> I am encountering a reproducible PCIe/EEH regression on a PowerNV
> platform (Talos II, POWER9) when moving from kernel 6.18.x to the
> 7.1.x series.
>
> Hardware Context:
>
> Platform: Raptor Computing Systems Talos II (PowerNV, POWER9)
>
> Storage Controller: 0001:01:00.0 SATA controller: JMicron Technology
> Corp. JMB58x AHCI SATA controller (direct root port, no PLX bridge)
>
> The Issue: On mainline/Arch POWER kernel 7.1.4, the kernel's
> enforcement of a 32-bit DMA mask (AHCI_HFLAG_32BIT_ONLY) on the JMB58x
> controller appears to be causing a low-memory translation failure
> during early initialization. This immediately trips the PowerNV Host
> Bridge (PHB) into an Enhanced Error Handling (EEH) frozen state and
> locks up the bus.
>
> Conversely, booting the exact same hardware setup with a 6.18.7 kernel
> succeeds completely, as it negotiates the 64-bit DMA window without
> triggering the platform translation fault.
>
> Boot Log Contrast:
>
> Under working 6.18.7 kernel:
>
> # dmesg | grep -iE "ahci|eeh"

1. Could you please attach and share the full dmesg (with no grep
   filtering) for both 6.18.7 (working kernel) and 7.1.4 (non-working
   kernel) along with their respective .config files. That should
   clarify few other things.

2. cat /sys/firmware/opal/msglog
   From both the kernels (after bootup and possibly after an EEH error
   has occurred at-least once on the non-working kernel)

-ritesh


> [    0.211910] EEH: PowerNV platform initialized
> [    0.391733] EEH: Capable adapter found: recovery enabled.
> [    2.192909] ahci 0001:01:00.0: enabling device (0540 -> 0542)
> [    2.193223] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
> [    2.193264] ahci 0001:01:00.0: AHCI vers 0001.0301, 32 command
> slots, 6 Gbps, SATA mode
> [    2.193270] ahci 0001:01:00.0: 5/5 ports implemented (port mask 0x1f)
> [    2.193275] ahci 0001:01:00.0: flags: 64bit ncq sntf stag pm led
> clo pmp fbs pio slum part ccc apst boh
> [    2.196895] scsi host0: ahci
> [    2.197276] scsi host1: ahci
> [    2.198768] scsi host2: ahci
> [    2.199068] scsi host3: ahci
> [    2.199995] scsi host4: ahci
>
> Under broken 7.1.4 kernel:
>
>  # dmesg | grep -iE "ahci|eeh"
>  [    3.298353] EEH: PowerNV platform initialized
>  [    3.460692] EEH: Capable adapter found: recovery enabled.
>  [    4.510396] ahci 0001:01:00.0: enabling device (0540 -> 0542)
>  [    4.510428] ahci 0001:01:00.0: controller can't do 64bit DMA, forcing 32bit
>  [    4.510691] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
>  [    4.510735] ahci 0001:01:00.0: AHCI vers 0001.0301, 32 command
> slots, 6 Gbps, SATA mode
>  [    4.510742] ahci 0001:01:00.0: 5/5 ports implemented (port mask 0x1f)
>  [    4.510747] ahci 0001:01:00.0: flags: ncq sntf stag pm led clo pmp
> fbs pio slum part ccc apst boh
>  [    4.512167] scsi host0: ahci
>  [    4.512556] scsi host1: ahci
>  [    4.512831] scsi host2: ahci
>  [    4.513076] scsi host3: ahci
>  [    4.513466] scsi host4: ahci
>  [    4.824011] EEH: Recovering PHB#1-PE#fd
>  [    4.824019] EEH: PE location: UOPWR.A100019-Node0-CPU1 Slot1 (8x),
> PHB location: N/A
>  [    4.824023] EEH: Frozen PHB#1-PE#fd detected
>  [    4.824025] EEH: Call Trace:
>  [    4.824027] EEH: [000000007ad80bda] __eeh_send_failure_event+0x68/0x140
>  [    4.824038] EEH: [0000000042fa5ce7] eeh_dev_check_failure.part.0+0x280/0x6d0
>  [    4.824044] EEH: [00000000ca5dc769] ioread32+0xd0/0x190
>  [    4.824051] EEH: [000000000ad730be] ata_wait_register+0xa4/0x108 [libata]
>  [    4.824064] EEH: [000000007fb91492]
> ahci_exec_polled_cmd.constprop.0+0x104/0x210 [libahci]
>  [    4.824070] EEH: [0000000063044e17] ahci_do_softreset+0x140/0x380 [libahci]
>  [    4.824074] EEH: [00000000d97bc211]
> ata_do_reset.constprop.0+0xbc/0xf0 [libata]
>  [    4.824084] EEH: [0000000093546eba] ata_eh_reset+0xb14/0x12d0 [libata]
>  [    4.824094] EEH: [00000000b2341f96] ata_eh_recover+0x3f0/0x1580 [libata]
>  [    4.824102] EEH: [00000000bbcd8f21]
> sata_pmp_error_handler+0x388/0xb90 [libata]
>  [    4.824109] EEH: [00000000b87225ad] ahci_error_handler+0x6c/0x100 [libahci]
>  [    4.824114] EEH: [00000000eda810ea]
> ata_scsi_port_error_handler+0x4a8/0x9a0 [libata]
>  [    4.824123] EEH: [0000000006f6b52d] ata_scsi_error+0xc0/0x130 [libata]
>  [    4.824131] EEH: [000000002528ea40] scsi_error_handler+0x11c/0x760
> [scsi_mod]
>  [    4.824142] EEH: [000000005ba0d15d] kthread+0x17c/0x190
>  [    4.824148] EEH: [0000000067c7c775] start_kernel_thread+0x14/0x18
>  [    4.824153] EEH: This PCI device has failed 1 times in the last
> hour and will be permanently disabled after 5 failures.
>  [    4.824156] EEH: Notify device drivers to shutdown
>  [    4.824159] EEH: Beginning: 'error_detected(IO frozen)'
>  [    4.824162] PCI 0001:01:00.0#00fd: EEH: driver not EEH aware
>  [    4.824165] EEH: Finished:'error_detected(IO frozen)' with
> aggregate recovery state:'none'
>  [    4.824170] EEH: Collect temporary log
>  [    4.824198] EEH: of node=0001:01:00.0
>  [    4.824203] EEH: PCI device/vendor: 0585197b
>  [    4.824207] EEH: PCI cmd/status register: 00100546
>  [    4.824209] EEH: PCI-E capabilities and status follow:
>  [    4.824220] EEH: PCI-E 00: 00120010 10008102 0000285e 0041a023
>  [    4.824230] EEH: PCI-E 10: 00230008 00000000 00000000 00000000
>  [    4.824232] EEH: PCI-E 20: 00000000
>  [    4.824234] EEH: PCI-E AER capability register set follows:
>  [    4.824245] EEH: PCI-E AER 00: 15020001 00000000 00400000 00462030
>  [    4.824254] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
>  [    4.824275] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
>  [    4.824279] EEH: PCI-E AER 30: 00000000 00000000
>  [    4.824313] EEH: Reset with hotplug activity
>  [   14.513778] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   14.513794] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.561995] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.591811] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.591821] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.591828] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.591834] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   15.591840] ahci 0001:01:00.0: AHCI controller unavailable!
>  [   20.592398] EEH: Sleep 5s ahead of complete hotplug
>  [   26.110843] ahci 0001:01:00.0: enabling device (0140 -> 0142)
>  [   26.110876] ahci 0001:01:00.0: controller can't do 64bit DMA, forcing 32bit
>  [   26.111136] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled
>
> this repeats until the device is disabled after 5 resets.
>
> Confirmation:
> I built 7.1.4 after reverting commit 105c4256 "ata: ahci: force 32-bit
> DMA for JMicron JMB582/JMB585" and boot completed successfully with an
> apparently working JMB585 sata controller.
>
> Conclusion:
> It appears that the recent JMicron 32-bit DMA workaround interacts
> poorly with PowerNV. Is there anything a user might do to work around
> this issue without reverting 105c4256?
>
> Let me know what additional information would be helpful.
>
> Thanks,
> Paul