Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Timothy Pearson <tpearson@raptorengineering.com>]

----- Original Message -----
> From: "Ritesh Harjani" <ritesh.list@gmail.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>, "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
> <gbatra@linux.ibm.com>
> Sent: Saturday, August 8, 2026 12:09:24 AM
> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

> Timothy Pearson <tpearson@raptorengineering.com> writes:
> 
>> ----- Original Message -----
>>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "linuxppc-dev"
>>> <linuxppc-dev@lists.ozlabs.org>
>>> Cc: "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
>>> <gbatra@linux.ibm.com>
>>> Sent: Thursday, August 6, 2026 12:27:22 AM
>>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>>
>>> Hi Timothy,
>>> 
>>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>>> 
>>>> On Linux 6.18, and potentially earlier Linux versions, attempting to pass
>>>> through any PCI device via VFIO that requires 32-bit DMA results in the
>>>> following error in the guest VM:
>>> 
>>> Can you confirm if you see the same issue on upstream kernel also or is
>>> it only till v6.18 kernel?
>>
>> I'll need to check, but I don't see why it would be limited to 6.18.  This used
>> to work, but it's fairly obvious IBM hasn't been testing or fixing any of the
>> VFIO or nested virtualization code for years now, especially on PowerNV.
>>
>>>>
>>>> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device
>>>> mask
>>>> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset:
>>>> 0x800000000000000
>>>>
>>>> The same devices work correctly on the host kernel.  It appears that somehow
>>>> 32-bit DMA is not available within a pSeries guest on a PowerNV host.
>>> 
>>> By any chance did you do kexec in the guest kernel? Is this issue only
>>> hitting with kexec or even with normal boot up of guest kernel is also
>>> showing up the same issue?
>>
>> Yes, we kexec().  This works on every other architecture, not sure why kexec()
>> would break it?
>>
>> In any case, from where I sit at this point it's IBM's responsibility as the
>> silicon vendor to ensure basic functionality works -- I don't see these issues
>> on amd64 or arm64, and frankly on this end we're tired of the constant lack of
>> QA from IBM causing every single kernel upgrade to be a game of Russian
>> Roulette.  IBM should be able to spin up a test machine easily enough to
>> reproduce, this isn't complex -- any card that needs 32-bit DMA is affected
>> including inexpensive ASmedia USB 3 controllers.  Pass it through to QEMU via
>> VFIO and watch it fail, it happens every time.
>>
> 
> I was trying to help by asking for basic help from you. If you report a
> problem upstream than we do expect such basic help e.g. is it only seen
> with kexec, logs etc, so that we can all save some time here.

No worries, appreciated.  This is only one of several major problems with Linux on PowerNV that we have run into after recent upgrades, and the primary focus at the moment is on restoring broken functionality.  In many cases, that means moving to different hardware, so gathering logs afterward can be difficult / impossible.

>>> Could you please share the dmesg logs for both guest and host kernel?
>>> That might give some more data about this issue.
>>
>> I don't have access to the setup at this point since it was already replaced
>> with something else to restore functionality.
> 
> I think this patch [1] might have fixed the issue that you reported and
> hence wanted to check if you only see the issue with kexec or also with
> reboots? And if you could have given it a try.
> 
> [1]: https://lore.kernel.org/all/20260803224029.60538-1-gbatra@linux.ibm.com/
> 
> But anyways - I have already asked others on that thread (who are expert
> of this area) to confirm if this patch could also fix the error you
> reported with 32-bit DMA devices.

Thank you for the link.  When / if we want to test a migration of that hardware back to VFIO on PowerNV I will ensure that patch is applied before testing.