[BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Timothy Pearson]

On Linux 6.18, and potentially earlier Linux versions, attempting to pass through any PCI device via VFIO that requires 32-bit DMA results in the following error in the guest VM:

[   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device mask
[   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset: 0x800000000000000

The same devices work correctly on the host kernel.  It appears that somehow 32-bit DMA is not available within a pSeries guest on a PowerNV host.

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Ritesh Harjani]

Hi Timothy,

Timothy Pearson <tpearson@raptorengineering.com> writes:

> On Linux 6.18, and potentially earlier Linux versions, attempting to pass through any PCI device via VFIO that requires 32-bit DMA results in the following error in the guest VM:

Can you confirm if you see the same issue on upstream kernel also or is
it only till v6.18 kernel?

>
> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device mask
> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset: 0x800000000000000
>
> The same devices work correctly on the host kernel.  It appears that somehow 32-bit DMA is not available within a pSeries guest on a PowerNV host.

By any chance did you do kexec in the guest kernel? Is this issue only
hitting with kexec or even with normal boot up of guest kernel is also
showing up the same issue?

Could you please share the dmesg logs for both guest and host kernel?
That might give some more data about this issue.

-ritesh

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Timothy Pearson]

----- Original Message -----
> From: "Ritesh Harjani" <ritesh.list@gmail.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>
> Cc: "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra" <gbatra@linux.ibm.com>
> Sent: Thursday, August 6, 2026 12:27:22 AM
> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

> Hi Timothy,
> 
> Timothy Pearson <tpearson@raptorengineering.com> writes:
> 
>> On Linux 6.18, and potentially earlier Linux versions, attempting to pass
>> through any PCI device via VFIO that requires 32-bit DMA results in the
>> following error in the guest VM:
> 
> Can you confirm if you see the same issue on upstream kernel also or is
> it only till v6.18 kernel?

I'll need to check, but I don't see why it would be limited to 6.18.  This used to work, but it's fairly obvious IBM hasn't been testing or fixing any of the VFIO or nested virtualization code for years now, especially on PowerNV.

>>
>> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device
>> mask
>> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset:
>> 0x800000000000000
>>
>> The same devices work correctly on the host kernel.  It appears that somehow
>> 32-bit DMA is not available within a pSeries guest on a PowerNV host.
> 
> By any chance did you do kexec in the guest kernel? Is this issue only
> hitting with kexec or even with normal boot up of guest kernel is also
> showing up the same issue?

Yes, we kexec().  This works on every other architecture, not sure why kexec() would break it?

In any case, from where I sit at this point it's IBM's responsibility as the silicon vendor to ensure basic functionality works -- I don't see these issues on amd64 or arm64, and frankly on this end we're tired of the constant lack of QA from IBM causing every single kernel upgrade to be a game of Russian Roulette.  IBM should be able to spin up a test machine easily enough to reproduce, this isn't complex -- any card that needs 32-bit DMA is affected including inexpensive ASmedia USB 3 controllers.  Pass it through to QEMU via VFIO and watch it fail, it happens every time.

> Could you please share the dmesg logs for both guest and host kernel?
> That might give some more data about this issue.

I don't have access to the setup at this point since it was already replaced with something else to restore functionality.

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Ritesh Harjani]

Timothy Pearson <tpearson@raptorengineering.com> writes:

> ----- Original Message -----
>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>
>> Cc: "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra" <gbatra@linux.ibm.com>
>> Sent: Thursday, August 6, 2026 12:27:22 AM
>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>
>> Hi Timothy,
>> 
>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>> 
>>> On Linux 6.18, and potentially earlier Linux versions, attempting to pass
>>> through any PCI device via VFIO that requires 32-bit DMA results in the
>>> following error in the guest VM:
>> 
>> Can you confirm if you see the same issue on upstream kernel also or is
>> it only till v6.18 kernel?
>
> I'll need to check, but I don't see why it would be limited to 6.18.  This used to work, but it's fairly obvious IBM hasn't been testing or fixing any of the VFIO or nested virtualization code for years now, especially on PowerNV.
>
>>>
>>> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device
>>> mask
>>> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset:
>>> 0x800000000000000
>>>
>>> The same devices work correctly on the host kernel.  It appears that somehow
>>> 32-bit DMA is not available within a pSeries guest on a PowerNV host.
>> 
>> By any chance did you do kexec in the guest kernel? Is this issue only
>> hitting with kexec or even with normal boot up of guest kernel is also
>> showing up the same issue?
>
> Yes, we kexec().  This works on every other architecture, not sure why kexec() would break it?
>
> In any case, from where I sit at this point it's IBM's responsibility as the silicon vendor to ensure basic functionality works -- I don't see these issues on amd64 or arm64, and frankly on this end we're tired of the constant lack of QA from IBM causing every single kernel upgrade to be a game of Russian Roulette.  IBM should be able to spin up a test machine easily enough to reproduce, this isn't complex -- any card that needs 32-bit DMA is affected including inexpensive ASmedia USB 3 controllers.  Pass it through to QEMU via VFIO and watch it fail, it happens every time.
>

I was trying to help by asking for basic help from you. If you report a
problem upstream than we do expect such basic help e.g. is it only seen
with kexec, logs etc, so that we can all save some time here.

>> Could you please share the dmesg logs for both guest and host kernel?
>> That might give some more data about this issue.
>
> I don't have access to the setup at this point since it was already replaced with something else to restore functionality.

I think this patch [1] might have fixed the issue that you reported and
hence wanted to check if you only see the issue with kexec or also with
reboots? And if you could have given it a try.

[1]: https://lore.kernel.org/all/20260803224029.60538-1-gbatra@linux.ibm.com/

But anyways - I have already asked others on that thread (who are expert
of this area) to confirm if this patch could also fix the error you
reported with 32-bit DMA devices.

-ritesh

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Timothy Pearson]

----- Original Message -----
> From: "Ritesh Harjani" <ritesh.list@gmail.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>, "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
> <gbatra@linux.ibm.com>
> Sent: Saturday, August 8, 2026 12:09:24 AM
> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

> Timothy Pearson <tpearson@raptorengineering.com> writes:
> 
>> ----- Original Message -----
>>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "linuxppc-dev"
>>> <linuxppc-dev@lists.ozlabs.org>
>>> Cc: "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
>>> <gbatra@linux.ibm.com>
>>> Sent: Thursday, August 6, 2026 12:27:22 AM
>>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>>
>>> Hi Timothy,
>>> 
>>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>>> 
>>>> On Linux 6.18, and potentially earlier Linux versions, attempting to pass
>>>> through any PCI device via VFIO that requires 32-bit DMA results in the
>>>> following error in the guest VM:
>>> 
>>> Can you confirm if you see the same issue on upstream kernel also or is
>>> it only till v6.18 kernel?
>>
>> I'll need to check, but I don't see why it would be limited to 6.18.  This used
>> to work, but it's fairly obvious IBM hasn't been testing or fixing any of the
>> VFIO or nested virtualization code for years now, especially on PowerNV.
>>
>>>>
>>>> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device
>>>> mask
>>>> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset:
>>>> 0x800000000000000
>>>>
>>>> The same devices work correctly on the host kernel.  It appears that somehow
>>>> 32-bit DMA is not available within a pSeries guest on a PowerNV host.
>>> 
>>> By any chance did you do kexec in the guest kernel? Is this issue only
>>> hitting with kexec or even with normal boot up of guest kernel is also
>>> showing up the same issue?
>>
>> Yes, we kexec().  This works on every other architecture, not sure why kexec()
>> would break it?
>>
>> In any case, from where I sit at this point it's IBM's responsibility as the
>> silicon vendor to ensure basic functionality works -- I don't see these issues
>> on amd64 or arm64, and frankly on this end we're tired of the constant lack of
>> QA from IBM causing every single kernel upgrade to be a game of Russian
>> Roulette.  IBM should be able to spin up a test machine easily enough to
>> reproduce, this isn't complex -- any card that needs 32-bit DMA is affected
>> including inexpensive ASmedia USB 3 controllers.  Pass it through to QEMU via
>> VFIO and watch it fail, it happens every time.
>>
> 
> I was trying to help by asking for basic help from you. If you report a
> problem upstream than we do expect such basic help e.g. is it only seen
> with kexec, logs etc, so that we can all save some time here.

No worries, appreciated.  This is only one of several major problems with Linux on PowerNV that we have run into after recent upgrades, and the primary focus at the moment is on restoring broken functionality.  In many cases, that means moving to different hardware, so gathering logs afterward can be difficult / impossible.

>>> Could you please share the dmesg logs for both guest and host kernel?
>>> That might give some more data about this issue.
>>
>> I don't have access to the setup at this point since it was already replaced
>> with something else to restore functionality.
> 
> I think this patch [1] might have fixed the issue that you reported and
> hence wanted to check if you only see the issue with kexec or also with
> reboots? And if you could have given it a try.
> 
> [1]: https://lore.kernel.org/all/20260803224029.60538-1-gbatra@linux.ibm.com/
> 
> But anyways - I have already asked others on that thread (who are expert
> of this area) to confirm if this patch could also fix the error you
> reported with 32-bit DMA devices.

Thank you for the link.  When / if we want to test a migration of that hardware back to VFIO on PowerNV I will ensure that patch is applied before testing.

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Ritesh Harjani]

Timothy Pearson <tpearson@raptorengineering.com> writes:

> ----- Original Message -----
>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>, "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
>> <gbatra@linux.ibm.com>
>> Sent: Saturday, August 8, 2026 12:09:24 AM
>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>
>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>> 
>>> ----- Original Message -----
>>>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "linuxppc-dev"
>>>> <linuxppc-dev@lists.ozlabs.org>
>>>> Cc: "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
>>>> <gbatra@linux.ibm.com>
>>>> Sent: Thursday, August 6, 2026 12:27:22 AM
>>>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>>>
>>>> Hi Timothy,
>>>> 
>>>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>>>> 
>>>>> On Linux 6.18, and potentially earlier Linux versions, attempting to pass
>>>>> through any PCI device via VFIO that requires 32-bit DMA results in the
>>>>> following error in the guest VM:
>>>> 
>>>> Can you confirm if you see the same issue on upstream kernel also or is
>>>> it only till v6.18 kernel?
>>>
>>> I'll need to check, but I don't see why it would be limited to 6.18.  This used
>>> to work, but it's fairly obvious IBM hasn't been testing or fixing any of the
>>> VFIO or nested virtualization code for years now, especially on PowerNV.
>>>
>>>>>
>>>>> [   11.705413] cx23885 0000:00:02.0: Warning: IOMMU offset too big for device
>>>>> mask
>>>>> [   11.709245] cx23885 0000:00:02.0: mask: 0xffffffff, table offset:
>>>>> 0x800000000000000
>>>>>
>>>>> The same devices work correctly on the host kernel.  It appears that somehow
>>>>> 32-bit DMA is not available within a pSeries guest on a PowerNV host.
>>>> 
>>>> By any chance did you do kexec in the guest kernel? Is this issue only
>>>> hitting with kexec or even with normal boot up of guest kernel is also
>>>> showing up the same issue?
>>>
>>> Yes, we kexec().  This works on every other architecture, not sure why kexec()
>>> would break it?
>>>
>>> In any case, from where I sit at this point it's IBM's responsibility as the
>>> silicon vendor to ensure basic functionality works -- I don't see these issues
>>> on amd64 or arm64, and frankly on this end we're tired of the constant lack of
>>> QA from IBM causing every single kernel upgrade to be a game of Russian
>>> Roulette.  IBM should be able to spin up a test machine easily enough to
>>> reproduce, this isn't complex -- any card that needs 32-bit DMA is affected
>>> including inexpensive ASmedia USB 3 controllers.  Pass it through to QEMU via
>>> VFIO and watch it fail, it happens every time.
>>>
>> 
>> I was trying to help by asking for basic help from you. If you report a
>> problem upstream than we do expect such basic help e.g. is it only seen
>> with kexec, logs etc, so that we can all save some time here.
>
> No worries, appreciated.  This is only one of several major problems with Linux on PowerNV that we have run into after recent upgrades, and the primary focus at the moment is on restoring broken functionality.  In many cases, that means moving to different hardware, so gathering logs afterward can be difficult / impossible.
>

Sorry to hear that you have been facing such issues recently. I will
bring this up internally. We will try to add more test matrix for
upstream PowerNV (note that we already have PowerNV tested regularly as
part of our upstream CI, but I suppose we could extend more KVM guest
testing there), so that we can reduce down on such reports.

>>>> Could you please share the dmesg logs for both guest and host kernel?
>>>> That might give some more data about this issue.
>>>
>>> I don't have access to the setup at this point since it was already replaced
>>> with something else to restore functionality.
>> 
>> I think this patch [1] might have fixed the issue that you reported and
>> hence wanted to check if you only see the issue with kexec or also with
>> reboots? And if you could have given it a try.
>> 
>> [1]: https://lore.kernel.org/all/20260803224029.60538-1-gbatra@linux.ibm.com/
>> 
>> But anyways - I have already asked others on that thread (who are expert
>> of this area) to confirm if this patch could also fix the error you
>> reported with 32-bit DMA devices.

Although not with VFIO - but I verified this issue by adding e1000
32-bit DMA device on the same PHB where other 64bit DMA devices were
attached (in my Qemu development environment).

And with that I was able to re-create the issue with kexec i.e. on kexec
I see the following error similar to what you reported in your
environment:

[    5.537726] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000 20000000 returned 0, lb=2000000000 ps=107 wn=0
[    5.540233] pci 0000:00:01.0: Adding to iommu group 0
[    5.556968] pci 0000:00:03.0: Adding to iommu group 0
[    5.567164] PCI: Probing PCI hardware done
[    9.395499] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct mapping: 0
[    9.396111] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct mapping: 0
[   13.228575] e1000: Intel(R) PRO/1000 Network Driver
[   13.229566] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   13.253804] e1000 0000:00:01.0: Warning: IOMMU offset too big for device mask
[   13.255200] e1000 0000:00:01.0: mask: 0xffffffff, table offset: 0x800000000000000
[   13.256611] e1000: No usable DMA config, aborting
[   13.267336] e1000 0000:00:01.0: probe with driver e1000 failed with error -5

This is failing with same error as you had reported:
     "Warning: IOMMU offset too big for device mask"

Whereas after I applied the patch mentioned in [1], I am able to see the
device working fine after kexec. Here are the logs with the fix applied:

[    6.197623] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000 20000000 returned 0, lb=2000000000 ps=107 wn=0
[    6.201686] pci 0000:00:01.0: Adding to iommu group 0
[    6.221890] pci 0000:00:03.0: Adding to iommu group 0
[    8.991990] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct mapping: 0
[    8.992710] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct mapping: 0
[   11.315607] e1000: Intel(R) PRO/1000 Network Driver
[   11.316191] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   11.671185] e1000 0000:00:01.0 eth1: (PCI:33MHz:32-bit) 52:54:00:12:34:56
[   11.672604] e1000 0000:00:01.0 eth1: Intel(R) PRO/1000 Network Connection

>
> Thank you for the link.  When / if we want to test a migration of that hardware back to VFIO on PowerNV I will ensure that patch is applied before testing.

Sure, I think this should fix the issue you reported. We have tagged
that fix with stable - so we will ensure it is backported to stable
kernel releases as well.

-ritesh

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Timothy Pearson]

----- Original Message -----
> From: "Ritesh Harjani" <ritesh.list@gmail.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>, "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
> <gbatra@linux.ibm.com>, "Madhavan Srinivasan" <maddy@linux.ibm.com>, "Venkat Rao Bagalkote" <venkat88@linux.ibm.com>
> Sent: Saturday, August 8, 2026 9:13:09 AM
> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

> Timothy Pearson <tpearson@raptorengineering.com> writes:
> 
>> ----- Original Message -----
>> No worries, appreciated.  This is only one of several major problems with Linux
>> on PowerNV that we have run into after recent upgrades, and the primary focus
>> at the moment is on restoring broken functionality.  In many cases, that means
>> moving to different hardware, so gathering logs afterward can be difficult /
>> impossible.
>>
> 
> Sorry to hear that you have been facing such issues recently. I will
> bring this up internally. We will try to add more test matrix for
> upstream PowerNV (note that we already have PowerNV tested regularly as
> part of our upstream CI, but I suppose we could extend more KVM guest
> testing there), so that we can reduce down on such reports.

That would be very helpful.  I have a number of other internal reports (and some external customer reports) that I will try to get together for submission here.  Given the testing on PowerNV seems quite sparse at the moment, if you do need additional PowerNV machines for testing we have bare metal cloud leases available.

One of the customer reports I already forwarded up to Github, but further logs aren't available as the customer has already moved to different hardware:

https://github.com/tbsdtv/linux_media/issues/431

The final straw that caused migration on that side was yet another VFIO bug on top of the invalid DMA that prevented EEH recovery and was forcing host resets to thaw the frozen PE.  I'll try to submit that report here as well.

> Although not with VFIO - but I verified this issue by adding e1000
> 32-bit DMA device on the same PHB where other 64bit DMA devices were
> attached (in my Qemu development environment).
> 
> And with that I was able to re-create the issue with kexec i.e. on kexec
> I see the following error similar to what you reported in your
> environment:
> 
> [    5.537726] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000
> 20000000 returned 0, lb=2000000000 ps=107 wn=0
> [    5.540233] pci 0000:00:01.0: Adding to iommu group 0
> [    5.556968] pci 0000:00:03.0: Adding to iommu group 0
> [    5.567164] PCI: Probing PCI hardware done
> [    9.395499] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
> mapping: 0
> [    9.396111] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
> mapping: 0
> [   13.228575] e1000: Intel(R) PRO/1000 Network Driver
> [   13.229566] e1000: Copyright (c) 1999-2006 Intel Corporation.
> [   13.253804] e1000 0000:00:01.0: Warning: IOMMU offset too big for device mask
> [   13.255200] e1000 0000:00:01.0: mask: 0xffffffff, table offset:
> 0x800000000000000
> [   13.256611] e1000: No usable DMA config, aborting
> [   13.267336] e1000 0000:00:01.0: probe with driver e1000 failed with error -5
> 
> This is failing with same error as you had reported:
>     "Warning: IOMMU offset too big for device mask"
> 
> Whereas after I applied the patch mentioned in [1], I am able to see the
> device working fine after kexec. Here are the logs with the fix applied:
> 
> [    6.197623] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000
> 20000000 returned 0, lb=2000000000 ps=107 wn=0
> [    6.201686] pci 0000:00:01.0: Adding to iommu group 0
> [    6.221890] pci 0000:00:03.0: Adding to iommu group 0
> [    8.991990] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
> mapping: 0
> [    8.992710] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
> mapping: 0
> [   11.315607] e1000: Intel(R) PRO/1000 Network Driver
> [   11.316191] e1000: Copyright (c) 1999-2006 Intel Corporation.
> [   11.671185] e1000 0000:00:01.0 eth1: (PCI:33MHz:32-bit) 52:54:00:12:34:56
> [   11.672604] e1000 0000:00:01.0 eth1: Intel(R) PRO/1000 Network Connection
> 
>>
>> Thank you for the link.  When / if we want to test a migration of that hardware
>> back to VFIO on PowerNV I will ensure that patch is applied before testing.
> 
> Sure, I think this should fix the issue you reported. We have tagged
> that fix with stable - so we will ensure it is backported to stable
> kernel releases as well.

That sounds great, thanks.  Does IBM have any plans to support Debian LTS (Freexian) so these patches can flow into the broken distribution kernels?

Thanks!

Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA

[Ritesh Harjani]

Hi Timothy

Timothy Pearson <tpearson@raptorengineering.com> writes:

> ----- Original Message -----
>> From: "Ritesh Harjani" <ritesh.list@gmail.com>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>
>> Cc: "linuxppc-dev" <linuxppc-dev@lists.ozlabs.org>, "Shivaprasad G Bhat" <sbhat@linux.ibm.com>, "Gaurav Batra"
>> <gbatra@linux.ibm.com>, "Madhavan Srinivasan" <maddy@linux.ibm.com>, "Venkat Rao Bagalkote" <venkat88@linux.ibm.com>
>> Sent: Saturday, August 8, 2026 9:13:09 AM
>> Subject: Re: [BUG] VFIO on POWER9 / QEMU fails to allocate 32-bit DMA
>
>> Timothy Pearson <tpearson@raptorengineering.com> writes:
>> 
>>> ----- Original Message -----
>>> No worries, appreciated.  This is only one of several major problems with Linux
>>> on PowerNV that we have run into after recent upgrades, and the primary focus
>>> at the moment is on restoring broken functionality.  In many cases, that means
>>> moving to different hardware, so gathering logs afterward can be difficult /
>>> impossible.
>>>
>> 
>> Sorry to hear that you have been facing such issues recently. I will
>> bring this up internally. We will try to add more test matrix for
>> upstream PowerNV (note that we already have PowerNV tested regularly as
>> part of our upstream CI, but I suppose we could extend more KVM guest
>> testing there), so that we can reduce down on such reports.
>
> That would be very helpful.  I have a number of other internal reports (and some external customer reports) that I will try to get together for submission here.  Given the testing on PowerNV seems quite sparse at the moment, if you do need additional PowerNV machines for testing we have bare metal cloud leases available.
>

Thanks for bringing up your concern. Based on your feedback, we brought
up this topic internally. So currently w.r.t upstream maintainance work,
we have build, boot, selftests and few other test runs, on our PowerNV
systems, but I agree our other major testing was happening on Pseries
platform. So, we decided to add more PowerNV machines to our internal CI
and expand / increase our upstream CI tests and runs on these baremetal
machines too. This is going to happen very soon, so hopefully, this
should reduce such reports in the future.


BTW - notice that the current issue - I reproduced on Pseries
development environment itself. So this one seems to be some special
case where you either have a 32-bit and a 64-bit HW sitting on the same
PE (partitionable endpoint) behind the same iommu group (like sometimes
behind a PCIe switch) or maybe the HW does both 32-bit and 64-bit probe.
But anyways the patch I pointed should fix that problem and we will
ensure it is backported to all stable kernels, and likely the
distributions should pick that up as well (we will take care of that too).

> One of the customer reports I already forwarded up to Github, but further logs aren't available as the customer has already moved to different hardware:
>
> https://github.com/tbsdtv/linux_media/issues/431
>
> The final straw that caused migration on that side was yet another VFIO bug on top of the invalid DMA that prevented EEH recovery and was forcing host resets to thaw the frozen PE.  I'll try to submit that report here as well.
>

Sure thanks. I see you already have shared few reports. We will take a
look at those. If there is anything else please feel free to share it
with us.

>> Although not with VFIO - but I verified this issue by adding e1000
>> 32-bit DMA device on the same PHB where other 64bit DMA devices were
>> attached (in my Qemu development environment).
>> 
>> And with that I was able to re-create the issue with kexec i.e. on kexec
>> I see the following error similar to what you reported in your
>> environment:
>> 
>> [    5.537726] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000
>> 20000000 returned 0, lb=2000000000 ps=107 wn=0
>> [    5.540233] pci 0000:00:01.0: Adding to iommu group 0
>> [    5.556968] pci 0000:00:03.0: Adding to iommu group 0
>> [    5.567164] PCI: Probing PCI hardware done
>> [    9.395499] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
>> mapping: 0
>> [    9.396111] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
>> mapping: 0
>> [   13.228575] e1000: Intel(R) PRO/1000 Network Driver
>> [   13.229566] e1000: Copyright (c) 1999-2006 Intel Corporation.
>> [   13.253804] e1000 0000:00:01.0: Warning: IOMMU offset too big for device mask
>> [   13.255200] e1000 0000:00:01.0: mask: 0xffffffff, table offset:
>> 0x800000000000000
>> [   13.256611] e1000: No usable DMA config, aborting
>> [   13.267336] e1000 0000:00:01.0: probe with driver e1000 failed with error -5
>> 
>> This is failing with same error as you had reported:
>>     "Warning: IOMMU offset too big for device mask"
>> 
>> Whereas after I applied the patch mentioned in [1], I am able to see the
>> device working fine after kexec. Here are the logs with the fix applied:
>> 
>> [    6.197623] pci 0000:00:01.0: ibm,query-pe-dma-windows(2026) 800 8000000
>> 20000000 returned 0, lb=2000000000 ps=107 wn=0
>> [    6.201686] pci 0000:00:01.0: Adding to iommu group 0
>> [    6.221890] pci 0000:00:03.0: Adding to iommu group 0
>> [    8.991990] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
>> mapping: 0
>> [    8.992710] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: 0, direct
>> mapping: 0
>> [   11.315607] e1000: Intel(R) PRO/1000 Network Driver
>> [   11.316191] e1000: Copyright (c) 1999-2006 Intel Corporation.
>> [   11.671185] e1000 0000:00:01.0 eth1: (PCI:33MHz:32-bit) 52:54:00:12:34:56
>> [   11.672604] e1000 0000:00:01.0 eth1: Intel(R) PRO/1000 Network Connection
>> 
>>>
>>> Thank you for the link.  When / if we want to test a migration of that hardware
>>> back to VFIO on PowerNV I will ensure that patch is applied before testing.
>> 
>> Sure, I think this should fix the issue you reported. We have tagged
>> that fix with stable - so we will ensure it is backported to stable
>> kernel releases as well.
>
> That sounds great, thanks.  Does IBM have any plans to support Debian LTS (Freexian) so these patches can flow into the broken distribution kernels?
>

yes, we do support that. Once the patch is merged and available in
stable kernels, we will ensure this is backported to this distribution
as well.

> Thanks!

I see you have already shared few more reports with us. We will be
looking into those and will bring those to closure too.

Thanks again for sharing the details with us.

-ritesh