Problem with PCI-passthrough to PV guest

Problem with PCI-passthrough to PV guest

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 3057 bytes --]

SUSE QA is seeing a problem with PCI-passthrough of a SR-IOV to a PV guest
running a 6.4 based kernel, but I can reproduce the problem with upstream
kernel, too.

The guest is configured with "e820_host=1", but the PCI region on the host
isn't marked as "reserved" in the memory map, so it won't be reserved in
the either.

As the guest doesn't have the ACPI table available describing the PCI regions,
/proc/iomem in the guest won't include those regions as used, resulting in
the kernel's resource management to use those areas for potential memory:

  resource: avoiding allocation from e820 entry [mem 0x78edc000-0x79868fff]
  resource: avoiding allocation from e820 entry [mem 0x79d2a000-0x8fffffff]
  resource: remaining [mem 0x0000000090000000-0x00000000ffdfffff] available
  resource: avoiding allocation from e820 entry [mem 0xc7ffc000-0xc7ffcfff]
  resource: remaining [mem 0x0000000090000000-0x00000000c7ffbfff] available

dom0 /proc/iomem:
  ...
  80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
  90000000-c7ffbfff : PCI Bus 0000:00    ← PCI MMIO window begins here
    90000000-900fffff : PCI Bus 0000:01  ← I350 VFs assigned in this range
    c6000000-c70fffff : PCI Bus 0000:04

dom0 e820 map:
  ...
  Xen: [mem 0x0000000079869000-0x0000000079d29fff] ACPI NVS
  Xen: [mem 0x0000000079d2a000-0x000000008fffffff] reserved
  Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
  Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
  ...

domU /proc/iomem:
  ...
  00100000-78f06fff : System RAM
    01000000-01ffffff : Kernel code
    ...
  90000000-97ffffff : System RAM
  fee00000-fee00fff : Local APIC

domU e820 map:
  ...
  Xen: [mem 0x0000000079869000-0x0000000079d29fff] ACPI NVS
  Xen: [mem 0x0000000079d2a000-0x000000008fffffff] reserved
  Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
  Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
  ...

The VF is showing up near 0x90000
  pci 0000:00:00.4: [8086:1520] type 00 class 0x020000
  pci 0000:00:00.4: reg 0x10: [mem 0x90004000-0x90007fff 64bit pref]
  pci 0000:00:00.4: reg 0x1c: [mem 0x90024000-0x90027fff 64bit pref]
  pcifront pci-0: New device on 0000:00:00.4 found.
  pcifront pci-0: claiming resource 0000:00:00.4/0
  pci 0000:00:00.4: can't claim BAR 0 [mem 0x90004000-0x90007fff 64bit pref]: 
address conflict with System RAM [mem 0x90000000-0x97ffffff]
  pcifront pci-0: Could not claim resource 0000:00:00.4/0! Device offline. Try 
using e820_host=1 in the guest config.
  pcifront pci-0: claiming resource 0000:00:00.4/3
  pci 0000:00:00.4: can't claim BAR 3 [mem 0x90024000-0x90027fff 64bit pref]: 
address conflict with System RAM [mem 0x90000000-0x97ffffff]
  pcifront pci-0: Could not claim resource 0000:00:00.4/3! Device offline. Try 
using e820_host=1 in the guest config.

My first idea for solving this was to add the PCI regions from dom0's
/proc/iomem to the e820 map of the guest, but this is more a hack than a sane
solution.

Thoughts?


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Problem with PCI-passthrough to PV guest

[Roger Pau Monné]

On Tue, May 05, 2026 at 05:53:31PM +0200, Jürgen Groß wrote:
> SUSE QA is seeing a problem with PCI-passthrough of a SR-IOV to a PV guest
> running a 6.4 based kernel, but I can reproduce the problem with upstream
> kernel, too.
> 
> The guest is configured with "e820_host=1", but the PCI region on the host
> isn't marked as "reserved" in the memory map, so it won't be reserved in
> the either.

But that's how it should be, device BARs shouldn't be in reserved
regions on the memory map (albeit we have seen this fairly often).

> As the guest doesn't have the ACPI table available describing the PCI regions,
> /proc/iomem in the guest won't include those regions as used, resulting in
> the kernel's resource management to use those areas for potential memory:
> 
>  resource: avoiding allocation from e820 entry [mem 0x78edc000-0x79868fff]
>  resource: avoiding allocation from e820 entry [mem 0x79d2a000-0x8fffffff]
>  resource: remaining [mem 0x0000000090000000-0x00000000ffdfffff] available
>  resource: avoiding allocation from e820 entry [mem 0xc7ffc000-0xc7ffcfff]
>  resource: remaining [mem 0x0000000090000000-0x00000000c7ffbfff] available
> 
> dom0 /proc/iomem:
>  ...
>  80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>  90000000-c7ffbfff : PCI Bus 0000:00    ← PCI MMIO window begins here
>    90000000-900fffff : PCI Bus 0000:01  ← I350 VFs assigned in this range
>    c6000000-c70fffff : PCI Bus 0000:04
> 
> dom0 e820 map:
>  ...
>  Xen: [mem 0x0000000079869000-0x0000000079d29fff] ACPI NVS
>  Xen: [mem 0x0000000079d2a000-0x000000008fffffff] reserved
>  Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
>  Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
>  ...
> 
> domU /proc/iomem:
>  ...
>  00100000-78f06fff : System RAM
>    01000000-01ffffff : Kernel code
>    ...
>  90000000-97ffffff : System RAM

But that's not in the mfn address space, it's just RAM in the pfn
space of the guest?

>  fee00000-fee00fff : Local APIC
> 
> domU e820 map:
>  ...
>  Xen: [mem 0x0000000079869000-0x0000000079d29fff] ACPI NVS
>  Xen: [mem 0x0000000079d2a000-0x000000008fffffff] reserved
>  Xen: [mem 0x00000000c7ffc000-0x00000000c7ffcfff] reserved
>  Xen: [mem 0x00000000fbffc000-0x00000000fbffcfff] reserved
>  ...
> 
> The VF is showing up near 0x90000
>  pci 0000:00:00.4: [8086:1520] type 00 class 0x020000
>  pci 0000:00:00.4: reg 0x10: [mem 0x90004000-0x90007fff 64bit pref]
>  pci 0000:00:00.4: reg 0x1c: [mem 0x90024000-0x90027fff 64bit pref]

While the above addresses are in the mfn address space?  I assume this
causes issues because MMIO is identity mapped in the pfn space.

>  pcifront pci-0: New device on 0000:00:00.4 found.
>  pcifront pci-0: claiming resource 0000:00:00.4/0
>  pci 0000:00:00.4: can't claim BAR 0 [mem 0x90004000-0x90007fff 64bit pref]:
> address conflict with System RAM [mem 0x90000000-0x97ffffff]
>  pcifront pci-0: Could not claim resource 0000:00:00.4/0! Device offline.
> Try using e820_host=1 in the guest config.
>  pcifront pci-0: claiming resource 0000:00:00.4/3
>  pci 0000:00:00.4: can't claim BAR 3 [mem 0x90024000-0x90027fff 64bit pref]:
> address conflict with System RAM [mem 0x90000000-0x97ffffff]
>  pcifront pci-0: Could not claim resource 0000:00:00.4/3! Device offline.
> Try using e820_host=1 in the guest config.
> 
> My first idea for solving this was to add the PCI regions from dom0's
> /proc/iomem to the e820 map of the guest, but this is more a hack than a sane
> solution.
> 
> Thoughts?

I think the issue is that the guest has created a pfn RAM range when
there is none in the provided e820.  When using "e820_host=1" the
guest should be limited to creating pfns only in the ranges marked as
RAM on the host e820 memory map.  Creating a pfn range over a hole in
the e820 shouldn't happen.

Regards, Roger.

Re: Problem with PCI-passthrough to PV guest

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 618 bytes --]

On 05.05.26 20:47, Roger Pau Monné wrote:
> On Tue, May 05, 2026 at 05:53:31PM +0200, Jürgen Groß wrote:
>> SUSE QA is seeing a problem with PCI-passthrough of a SR-IOV to a PV guest
>> running a 6.4 based kernel, but I can reproduce the problem with upstream
>> kernel, too.

Uh, seems my tested kernel was only "nearly upstream" (it was an early
7.0 kernel). It was missing commit 0949c646d646, which makes the difference.

So for now this issue will only show up for cases where a PV-guest is
setup for PCI passthrough and it gets memory hotplugged before the PCI
device is added to it.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Problem with PCI-passthrough to PV guest

[Roger Pau Monné]

On Wed, May 06, 2026 at 10:45:49AM +0200, Jürgen Groß wrote:
> On 05.05.26 20:47, Roger Pau Monné wrote:
> > On Tue, May 05, 2026 at 05:53:31PM +0200, Jürgen Groß wrote:
> > > SUSE QA is seeing a problem with PCI-passthrough of a SR-IOV to a PV guest
> > > running a 6.4 based kernel, but I can reproduce the problem with upstream
> > > kernel, too.
> 
> Uh, seems my tested kernel was only "nearly upstream" (it was an early
> 7.0 kernel). It was missing commit 0949c646d646, which makes the difference.
> 
> So for now this issue will only show up for cases where a PV-guest is
> setup for PCI passthrough and it gets memory hotplugged before the PCI
> device is added to it.

Hm, yes, this is yet more fallout from the unpopulated-alloc changes.
For PV guests that have the e820 host available we could try to create
the hotplug regions over what are host RAM region in the memory map,
as then we know we won't be overlapping with MMIO areas.

Thanks, Roger.

Re: Problem with PCI-passthrough to PV guest

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

On Wed, May 06, 2026 at 11:18:28AM +0200, Roger Pau Monné wrote:
> On Wed, May 06, 2026 at 10:45:49AM +0200, Jürgen Groß wrote:
> > On 05.05.26 20:47, Roger Pau Monné wrote:
> > > On Tue, May 05, 2026 at 05:53:31PM +0200, Jürgen Groß wrote:
> > > > SUSE QA is seeing a problem with PCI-passthrough of a SR-IOV to a PV guest
> > > > running a 6.4 based kernel, but I can reproduce the problem with upstream
> > > > kernel, too.
> > 
> > Uh, seems my tested kernel was only "nearly upstream" (it was an early
> > 7.0 kernel). It was missing commit 0949c646d646, which makes the difference.
> > 
> > So for now this issue will only show up for cases where a PV-guest is
> > setup for PCI passthrough and it gets memory hotplugged before the PCI
> > device is added to it.
> 
> Hm, yes, this is yet more fallout from the unpopulated-alloc changes.
> For PV guests that have the e820 host available we could try to create
> the hotplug regions over what are host RAM region in the memory map,
> as then we know we won't be overlapping with MMIO areas.

FWIW, my Linux testing series also(?) found a PV PCI-passthrough issue
in recent Linux (linux-next from a few days ago there). While it looks
to be a different issue, it would help a lot to have that series in, to
at least detect regressions early. See
https://lore.kernel.org/xen-devel/cover.30e6171ddf1c6a72eadf4af0a77c892d4f18d811.1777898148.git-series.marmarek@invisiblethingslab.com/

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]