Re: powerpc/pseries: EEH detected with xhci_hcd

[Sourabh Jain <sourabhjain@linux.ibm.com>]

On 07/08/26 14:27, Narayana Murty N wrote:
> HI Sourabh,
>
> On 19/07/26 9:59 AM, Sourabh Jain wrote:
>>
>>
>> On 16/07/26 10:21, Ritesh Harjani (IBM) wrote:
>>> Sourabh Jain <sourabhjain@linux.ibm.com> writes:
>>>
>>>> This issue only occurs with the Radix MMU.
>>>> Booting the same kernel with disable_radix=1 does not reproduce the 
>>>> issue.
>>>>
>>>> The system also has pmem.
>>> @Sourabh,
>>>
>>> Few requests -
>>> 1. Could you also attach the boot logs from the same system with the
>>>     same kernel but with disable_radix=1?
>>>
>>> 2. I can't really tell on how is MMU related to this issue? But I think
>>>     if you could share the working dmesg logs from Hash that might give
>>>     some more info. I am suspecting what if the firmware negotiates
>>>     different page sizes even for TCEs for Radix v/s Hash.
>>
>> Attached the dmesg of the kernel where issue is reproducible with and 
>> without
>> disable_radix=1
>>
>>>
>>> 3. Would it be possible for you to try git bisect and identify the
>>>     culprit commit?
>>
>> git bisect start
>> # status: waiting for both good and bad commits
>> # good: [1471c517cf7dae1a6342fb821d8ed501af956dd0] powerpc/iommu: 
>> bypass DMA APIs for coherent allocations for pre-mapped memory
>> git bisect good 1471c517cf7dae1a6342fb821d8ed501af956dd0
>> # status: waiting for bad commit, 1 good commit known
>> # bad: [328335a79487ec38d6b0e1aa807785b0f75e594d] 
>> powerpc/powernv/iommu: iommu incorrectly bypass DMA APIs
>> git bisect bad 328335a79487ec38d6b0e1aa807785b0f75e594d
>> # bad: [939faf71cf7ca9ab3d1bd2912ac0e203d4d7156a] Merge tag 'drm- 
>> next-2026-02-11' of https://gitlab.freedesktop.org/drm/kernel
>> git bisect bad 939faf71cf7ca9ab3d1bd2912ac0e203d4d7156a
>> # good: [66bbe4a8ed73f1187a4271c58f0ea30f42debe0d] Merge tag 'irq- 
>> core-2026-02-09' of 
>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>> git bisect good 66bbe4a8ed73f1187a4271c58f0ea30f42debe0d
>> # bad: [d061251387903e8502843ac983553f0b2e098ef8] Merge tag 
>> 'sound-7.0- rc1' of 
>> git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
>> git bisect bad d061251387903e8502843ac983553f0b2e098ef8
>> # good: [f51d34065de4f29fd237276a49cad06ac7356a55] Merge tag 
>> 'socfpga_dts_updates_for_v6.20_v3' of git://git.kernel.org/pub/scm/ 
>> linux/kernel/git/dinguyen/linux into soc/dt
>> git bisect good f51d34065de4f29fd237276a49cad06ac7356a55
>> # bad: [e86dda7bde8801d32ffe7d1570fe173cab14d1ba] Merge tag 
>> 'spi-v6.20' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
>> git bisect bad e86dda7bde8801d32ffe7d1570fe173cab14d1ba
>> # good: [bdbddf72a2ab1cfea699959795d70df3931eefe7] Merge tag 'soc- 
>> drivers-7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
>> git bisect good bdbddf72a2ab1cfea699959795d70df3931eefe7
>> # good: [8ea39d960c9f890e9213cdcfcbe4b3f281acd12f] spi: add 
>> multi-lane support
>> git bisect good 8ea39d960c9f890e9213cdcfcbe4b3f281acd12f
>> # bad: [d5cbd9f332c70be9589201474b9477baf9b5a24d] Merge tag 'regmap- 
>> v6.20' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
>> git bisect bad d5cbd9f332c70be9589201474b9477baf9b5a24d
>> # good: [8ad8d24d962804d3c54b5b7542c4cb7cd6bacb3b] Merge tag 'parisc- 
>> for-7.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/ 
>> parisc-linux
>> git bisect good 8ad8d24d962804d3c54b5b7542c4cb7cd6bacb3b
>> # bad: [1d72a02d65a733ad124bacc2db3fb90fa81a612b] docs: trusted- 
>> encryped: add PKWM as a new trust source
>> git bisect bad 1d72a02d65a733ad124bacc2db3fb90fa81a612b
>> # bad: [11d45eee9f42f7d3f9c9f190a6f79e570d345fe2] powerpc64/bpf: 
>> Additional NVR handling for bpf_throw
>> git bisect bad 11d45eee9f42f7d3f9c9f190a6f79e570d345fe2
>> # bad: [2ed2d8f6fb3853daa536d79e0799cbac91fac9f5] powerpc64/bpf: 
>> Support tailcalls with subprogs
>> git bisect bad 2ed2d8f6fb3853daa536d79e0799cbac91fac9f5
>> # bad: [815a8d2feb5615ae7f0b5befd206af0b0160614c] powerpc/eeh: fix 
>> recursive pci_lock_rescan_remove locking in EEH event handling
>> git bisect bad 815a8d2feb5615ae7f0b5befd206af0b0160614c
>>
>> Reverting commit 815a8d2feb56 ("powerpc/eeh: fix recursive 
>> pci_lock_rescan_remove locking in EEH event handling")
>> on top of commit fce2dfa773ced15f27d resolves the issue.
>>
>> - Sourabh Jain
>>
>>>
>>>
>>> Btw, to clarify I think there are two issues here:
>>> 1. EEH triggered for the USB adapter - I think you meant that with Hash
>>>     you don't even see EEH error being triggered at all.
>>>
>>> 2. Second problem is - EEH recovery causing this WARNING.
>>>
>>> [   17.375534] [    T118] WARNING: arch/powerpc/kernel/eeh_pe.c:439 
>>> at eeh_pe_tree_remove+0xf8/0x2f0, CPU#10: eehd/118
>>> <...>
>>> [   17.375599] [    T118] CPU: 10 UID: 0 PID: 118 Comm: eehd 
>>> Tainted: G            E       7.2.0-rc3upstream+ #3 PREEMPT(lazy)  
>>> f29942c1738362234fefd88de0d1ed711f1a557e
>>> [   17.375603] [    T118] Tainted: [E]=UNSIGNED_MODULE
>>> [   17.375604] [    T118] Hardware name: IBM,9080-HEX Power11 
>>> (architected) 0x820200 0xf000007 of:IBM,FW1120.00 (NH1120_135) 
>>> hv:phyp pSeries
>>> <...>
>>> [   17.375650] [    T118] NIP [c00000000004a048] 
>>> eeh_pe_tree_remove+0xf8/0x2f0
>>> [   17.375654] [    T118] LR [c00000000004ba6c] 
>>> eeh_pe_detach_dev+0x5c/0xa0
>>> [   17.375658] [    T118] Call Trace:
>>> [   17.375659] [    T118] [c00000000a357c90] [c00000000dd32800] 
>>> 0xc00000000dd32800 (unreliable)
>>> [   17.375668] [    T118] [c00000000a357d30] [c00000000004ba6c] 
>>> eeh_pe_detach_dev+0x5c/0xa0
>>> [   17.375672] [    T118] [c00000000a357d60] [c0000000000496ec] 
>>> eeh_pe_traverse+0x6c/0xf0
>>> [   17.375676] [    T118] [c00000000a357da0] [c00000000004e100] 
>>> eeh_reset_device+0x1f8/0x298
>>> [   17.375680] [    T118] [c00000000a357e50] [c00000000004ce5c] 
>>> eeh_handle_normal_event+0x61c/0xa60
>>> [   17.375685] [    T118] [c00000000a357f30] [c00000000004e298] 
>>> eeh_event_handler+0xf8/0x190
>>> [   17.375689] [    T118] [c00000000a357f90] [c00000000025dab4] 
>>> kthread+0x194/0x1b0
>>>
>>>
>>> -ritesh
>>>
> Thanks for the bisect output — that's very helpful.

>
> Commit 1010b4c012b0 ("powerpc/eeh: Make EEH driver device hotplug safe")
> restructured EEH driver synchronization with the PCI hotplug layer, but
> inadvertently introduced a deadlock that was silently preventing eehd 
> from detecting and recovering from EEH events.
>
> Commit 815a8d2feb56 ("powerpc/eeh: fix recursive pci_lock_rescan_remove
> locking in EEH event handling") resolved that deadlock — but as a
> consequence, EEH processing now proceeds further and exposes this
> pre-existing WARNING in eeh_pe_tree_remove().
>
> This means reverting 815a8d2feb56 makes the issue disappear only because
> it re-introduces the deadlock that was masking it — it is not a real fix.

Thanks for the clarification. So, commit 815a8d2feb56 is not the cause 
of the
problem; instead, it is making the issue visible.


>
> To further validate this, it would be useful to inject an error into the
> PE on a kernel with 815a8d2feb56 applied, and confirm whether EEH
> detection and recovery can proceed correctly end-to-end once the
> WARNING in eeh_pe_tree_remove() is also addressed.

Kernel booting with commit 815a8d2feb56 included exhausts the EEH recovery
limit before even reaching the console, so I’m not sure how injecting 
the error
again would help. What do you think?

Even if I try injecting the error, could you please explain how manual 
error injection
would help in this case?

FYI, this issue is not observed with the HASH MMU. Do you think the MMU 
has a role
here, or could it just be a timing issue?

- Sourabh Jain