Re: [BUG] PCIe passthrough of MCS9990 fails

[Narayana Murty N <nnmlinux@linux.ibm.com>]

Hi Timothy,

Thanks for the detailed logs.

On 19/09/26 12:19 AM, Timothy Pearson wrote:
> I've also seen the following failure in the guest when passing through more than one PCIe function:
> 
> [    0.710454] pci 0000:01:00.1: EHCI: unrecognized capability 00
> [    0.710546] EEH: Recovering PHB#0-PE#10001
> [    0.710598] EEH: PE location: N/A, PHB location: N/A
> [    0.710669] EEH: Frozen PHB#0-PE#10001 detected
> [    0.710745] EEH: Call Trace:
> [    0.710832] EEH: [000000007d519995] __eeh_send_failure_event+0x7c/0x160
> [    0.710944] EEH: [00000000419515d5] eeh_dev_check_failure+0x2e0/0x6d0
> [    0.711011] EEH: [000000008950df7e] quirk_usb_early_handoff+0xca4/0xfd0
> [    0.711076] EEH: [000000002528ae55] pci_fixup_device+0x140/0x2d0
> [    0.711137] EEH: [000000002e85a626] pci_apply_final_quirks+0xb4/0x1b8
> [    0.711199] EEH: [00000000e0915ba4] do_one_initcall+0x80/0x3b0
> [    0.711255] EEH: [0000000014cb0304] kernel_init_freeable+0x340/0x3e4
> [    0.711311] EEH: [00000000b18e64ad] kernel_init+0x30/0x1a0
> [    0.711357] EEH: [0000000047d4e5e3] ret_from_kernel_user_thread+0x14/0x1c
> [    0.711413] EEH: This PCI device has failed 1 times in the last hour and will be permanently disabled after 5 failures.
> [    0.711493] EEH: Notify device drivers to shutdown
> [    0.711538] EEH: Beginning: 'error_detected(IO frozen)'
> [    0.711584] PCI 0000:01:00.1#10001: EEH: no driver
> [    0.711589] PCI 0000:01:00.3#10001: EEH: no driver
> [    0.711644] PCI 0000:01:00.5#10001: EEH: no driver
> [    0.711689] PCI 0000:01:00.7#10001: EEH: no driver
> [    0.711737] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'none'
> [    0.711860] EEH: Collect temporary log
> [    0.712602] EEH: of node=0000:01:00.1
> [    0.712654] EEH: PCI device/vendor: 99909710
> [    0.712720] EEH: PCI cmd/status register: 00100146
> [    0.712761] EEH: PCI-E capabilities and status follow:
> [    0.712876] EEH: PCI-E 00: 00010010 00008fc1 0000283f 01078011
> [    0.712989] EEH: PCI-E 10: 00110008 00000000 000003c0 00000000
> [    0.713044] EEH: PCI-E 20: 00000000
> [    0.713082] EEH: of node=0000:01:00.3
> [    0.713129] EEH: PCI device/vendor: 99909710
> [    0.713187] EEH: PCI cmd/status register: 00100146
> [    0.713232] EEH: PCI-E capabilities and status follow:
> [    0.713345] EEH: PCI-E 00: 00010010 00008fc1 0000283f 01078011
> [    0.713460] EEH: PCI-E 10: 00110008 00000000 000003c0 00000000
> [    0.713513] EEH: PCI-E 20: 00000000
> [    0.713547] EEH: of node=0000:01:00.5
> [    0.713595] EEH: PCI device/vendor: 99909710
> [    0.713655] EEH: PCI cmd/status register: 00100146
> [    0.713700] EEH: PCI-E capabilities and status follow:
> [    0.713812] EEH: PCI-E 00: 00010010 00008fc1 0000283f 01078011
> [    0.713938] EEH: PCI-E 10: 00110008 00000000 000003c0 00000000
> [    0.714007] EEH: PCI-E 20: 00000000
> [    0.714052] EEH: of node=0000:01:00.7
> [    0.714108] EEH: PCI device/vendor: 99909710
> [    0.714182] EEH: PCI cmd/status register: 00100146
> [    0.714239] EEH: PCI-E capabilities and status follow:
> [    0.714371] EEH: PCI-E 00: 00010010 00008fc1 0000283f 01078011
> [    0.714501] EEH: PCI-E 10: 00110008 00000000 000003c0 00000000
> [    0.714571] EEH: PCI-E 20: 00000000
> [    0.714623] EEH: Reset with hotplug activity
> [    0.714942] pci 0000:01:00.1: quirk_usb_early_handoff+0x0/0xfd0 took 12340 usecs
> [    0.716273] PCI: CLS 0 bytes, default 128
> [    0.716424] Trying to unpack rootfs image as initramfs...
> [    0.725599] Initialise system trusted keyrings
> [    0.725679] Key type blacklist registered
> [    0.725797] workingset: timestamp_bits=38 max_order=19 bucket_order=0
> [    0.725891] zbud: loaded
> 
> Passing through one single EHCI function seems to at least allow the guest to boot, but only *once* after each power cycle.
> 
> All functions are in the same IOMMU group:
> 
> [  617.749136] pci 0031:01:00.0: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.749797] pci 0031:01:00.1: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.750412] pci 0031:01:00.2: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.751035] pci 0031:01:00.3: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.751668] pci 0031:01:00.4: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.752270] pci 0031:01:00.5: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.752871] pci 0031:01:00.6: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.753467] pci 0031:01:00.7: BAR 0 [mem size 0x00001000]: can't claim; no address assigned
> [  617.754039] pci 0031:00:00.0: disabling bridge window [io  0x0000-0xffffffffffffffff] to [bus 01] (unused)
> [  617.754642] pci 0031:00:00.0: disabling bridge window [mem 0x00000000-0xffffffffffffffff 64bit pref] to [bus 01] (unused)
> [  617.755257] pci 0031:01:00.0: BAR 0 [mem 0x620c080000000-0x620c080000fff]: assigned
> [  617.755813] pci 0031:01:00.1: BAR 0 [mem 0x620c080010000-0x620c080010fff]: assigned
> [  617.756195] pci 0031:01:00.2: BAR 0 [mem 0x620c080020000-0x620c080020fff]: assigned
> [  617.756559] pci 0031:01:00.3: BAR 0 [mem 0x620c080030000-0x620c080030fff]: assigned
> [  617.756923] pci 0031:01:00.4: BAR 0 [mem 0x620c080040000-0x620c080040fff]: assigned
> [  617.757281] pci 0031:01:00.5: BAR 0 [mem 0x620c080050000-0x620c080050fff]: assigned
> [  617.757635] pci 0031:01:00.6: BAR 0 [mem 0x620c080060000-0x620c080060fff]: assigned
> [  617.757980] pci 0031:01:00.7: BAR 0 [mem 0x620c080070000-0x620c080070fff]: assigned
> [  617.758332] pci 0031:00:00.0: PCI bridge to [bus 01]
> [  617.758662] pci 0031:00:00.0:   bridge window [mem 0x620c080000000-0x620c0ffefffff]
> [  617.759016] pci_bus 0031:01: Configuring PE for bus
> [  617.759357] pci 0031:01     : [PE# fd] Secondary bus 0x0000000000000001 associated with PE#fd
> [  617.759875] pci 0031:01:00.0: Configured PE#fd
> [  617.760213] pci 0031:01     : [PE# fd] Setting up 32-bit TCE table at 0..80000000
> [  617.762565] pci 0031:01     : [PE# fd] Setting up window#0 0..7ffffffffff pg=10000
> [  617.763014] pci 0031:01     : [PE# fd] Enabling 64-bit DMA bypass
> [  617.763631] pci 0031:01:00.0: Adding to iommu group 10
> [  617.764218] pci 0031:01:00.0: enabling device (0140 -> 0142)
> [  617.764735] pci 0031:01:00.1: Added to existing PE#fd
> [  617.765064] pci 0031:01:00.1: Adding to iommu group 10
> [  617.765483] pci 0031:01:00.1: enabling device (0140 -> 0142)
> [  617.766003] pci 0031:01:00.2: Added to existing PE#fd
> [  617.766331] pci 0031:01:00.2: Adding to iommu group 10
> [  617.766764] pci 0031:01:00.2: enabling device (0140 -> 0142)
> [  617.767179] pci 0031:01:00.3: Added to existing PE#fd
> [  617.767517] pci 0031:01:00.3: Adding to iommu group 10
> [  617.767932] pci 0031:01:00.3: enabling device (0140 -> 0142)
> [  617.768339] pci 0031:01:00.4: Added to existing PE#fd
> [  617.768655] pci 0031:01:00.4: Adding to iommu group 10
> [  617.769068] pci 0031:01:00.4: enabling device (0140 -> 0142)
> [  617.769468] pci 0031:01:00.5: Added to existing PE#fd
> [  617.769768] pci 0031:01:00.5: Adding to iommu group 10
> [  617.770176] pci 0031:01:00.5: enabling device (0140 -> 0142)
> [  617.770623] pci 0031:01:00.6: Added to existing PE#fd
> [  617.770921] pci 0031:01:00.6: Adding to iommu group 10
> [  617.771321] pci 0031:01:00.6: enabling device (0140 -> 0142)
> [  617.771729] pci 0031:01:00.7: Added to existing PE#fd
> [  617.772014] pci 0031:01:00.7: Adding to iommu group 10
> [  617.772451] pci 0031:01:00.7: enabling device (0140 -> 0142)
> 
> 
> ----- Original Message -----
>> From: "Timothy Pearson" <tpearson@raptorengineeringinc.com>
>> To: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>
>> Cc: "Ritesh Harjani" <ritesh.list@gmail.com>
>> Sent: Friday, September 18, 2026 1:32:51 PM
>> Subject: [BUG] PCIe passthrough of MCS9990 fails
> 
>> When passing through all eight PCIe functions of a MCS9990 PCIe USB
>> multifunction 1.0/2.0 controller on a PowerNV POWER9 host using qemu, the guest
>> fails to launch:
>>
>> qemu-system-ppc64: -device vfio-pci,host=0031:01:00.1,bus=bridge-1,addr=00.1:
>> vfio 0031:01:00.1: Failed to set up TRIGGER eventfd signaling for interrupt
>> INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
>>
>> If I instead pass through only one function, or only pass through odd functions
>> (the EHCI controller set), I get the following errors in the guest:
>>
>> [    0.886625] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002,
>> bcdDevice= 6.18
>> [    0.886710] usb usb1: New USB device strings: Mfr=3, Product=2,
>> SerialNumber=1
>> [    0.886791] usb usb1: Product: EHCI Host Controller
>> [    0.886845] usb usb1: Manufacturer: Linux 6.18.38 ehci_hcd
>> [    0.886902] usb usb1: SerialNumber: 0000:01:00.0
>> [    1.142469] usb 1-1: new high-speed USB device number 2 using ehci-pci
>> [    6.530490] usb 1-1: device descriptor read/64, error -110
>> [   22.147501] usb 1-1: device descriptor read/64, error -110
>> [   22.371536] usb 1-1: new high-speed USB device number 3 using ehci-pci
>> [   27.526896] usb 1-1: device descriptor read/64, error -110
>> [   43.141934] usb 1-1: device descriptor read/64, error -110
>> [   43.245985] usb usb1-port1: attempt power cycle
>> [   43.638173] usb 1-1: new high-speed USB device number 4 using ehci-pci
>> [   54.491254] usb 1-1: device not accepting address 4, error -110
>> [   54.611274] usb 1-1: new high-speed USB device number 5 using ehci-pci
>>
>> If I reboot the guest, SLOF fails as follows:
>>
>> Scanning USB
>>   EHCI: Initializing
>> usb-ehci: reset failed
>> usb-ehci: control transfer timed out_
>> usb-ehci: unable to setup device on port 0
>>
>> Note that this card works perfectly on the host kernel.
 From what I can tell so far, this looks like it may be related to the
passthrough device falling back to legacy INTx on a shared IRQ.

My current suspicion is that the multifunction device may not provide
usable per-function INTx masking, so VFIO cannot safely mask one
function while the IRQ is shared by the other functions. That could
explain the interrupt behaviour we are seeing, but I would like to
confirm the actual interrupt mode before drawing any conclusion.

Could you please share the following from the failing setup?

lspci -vvv for all functions of the multifunction device on the host
lspci -vvv for the passed-through device inside the guest
/proc/interrupts on the host before and after starting the guest
any VFIO/IRQ related messages from host dmesg
the relevant QEMU command line

In particular, I would like to confirm whether the device is actually
running with legacy INTx, whether the functions share the same host IRQ,
and whether MSI/MSI-X is being enabled successfully in the guest.

Thanks,
Narayana
>