xen/arm: Virtio-PCI for dom0less on ARM

xen/arm: Virtio-PCI for dom0less on ARM

[Edgar E. Iglesias]

Hi all,

Following up on the ARM virtio-pci series I posted a while back ago.

There have been some concerns around the delayed and silent apperance of
devices on the ECAM area. The spec is not super clear wether this is OK or
not but I'm providing some references to the PCI specs and to some real cases
where this is used for FPGAs.

There are two options how to implement virtio-pci that we've discussed:
1. VPCI + IOREQ
2. IOREQ only

There are pros and cons with both. For example with #1, has the benefit that
we would only have a single PCIe RC (in Xen) and we could emulate a hotplug
capable expansion port with a standard way to notify when PCI devices plug in.
Approach #2 has the benefit that there is (almost) no additional complexity
or code added to Xen, almost everything lives outside.
IMO, both options have merit and both could co-exist.

For dynamic xl flows, options #2 already works without modifications to Xen.
Users need to pass the correct command-line options to QEMU and a device-tree
fragment with the pci-generic-ecam-host device.

For static dom0less flows, we can do the same as for xl flows but we have the
additional problem of domU's PCI bus enumeration racing with QEMU.
On x86, when domU's access a memory range that has not yet got IOREQ's
connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
writes ignored. This makes it easy for guests to wait for IOREQ devices to
pop up since guests will find an empty bus and can initiate a rescan later
when QEMU attaches. On ARM, we trap on these accesses.

If we on ARM add support for MMIO background regions with a default read value,
i.e mmio handlers that have lower priority than IOREQs and that are
read-const + writes-ignored, we could support the same flow on ARM.
This may be generally useful for other devices as well (e.g virtio-mmio or
something else). We could also use this to defer PCI enumeration.

So the next versions of this series I was thinking to remove the PCI specifics
and instead add FDT bindings to ARM dom0less enabling setup of user
configurable (by address-range and read-value) background mmio regions.
Xen would then support option #2 without any PCI specifics added.

Thoughts?

Cheers,
Edgar

# References to spec

PCI express base specification:
7.5.1.1.1 Vendor ID Register (Offset 00h)
The Vendor ID register is HwInit and the value in this register identifies the manufacturer of the Function. In keeping with
PCI-SIG procedures, valid vendor identifiers must be allocated by the PCI-SIG to ensure uniqueness. Each vendor must
have at least one Vendor ID. It is recommended that software read the Vendor ID register to determine if a Function is
present, where a value of FFFFh indicates that no Function is present.

PCI Firmware Specification:
3.5 Device State at Firmware/Operating System Handoff
Page 34:
The operating system is required to configure PCI subsystems:
 During hotplug
 For devices that take too long to come out of reset
 PCI-to-PCI bridges that are at levels below what firmware is designed to configure

Page 36:
Note: The operating system does not have to walk all buses during boot. The kernel can
automatically configure devices on request; i.e., an event can cause a scan of I/O on demand.

FPGA's can be programmed at runtime and appear on the ECAM bus silently.
An PCI rescan needs to be triggered for the OS to discover the device:
Intel FPGAs:
https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html

Re: xen/arm: Virtio-PCI for dom0less on ARM

[Bertrand Marquis]

Hi Edgar,

> On 20 May 2025, at 10:04, Edgar E. Iglesias <edgar.iglesias@amd.com> wrote:
> 
> Hi all,
> 
> Following up on the ARM virtio-pci series I posted a while back ago.
> 
> There have been some concerns around the delayed and silent apperance of
> devices on the ECAM area. The spec is not super clear wether this is OK or
> not but I'm providing some references to the PCI specs and to some real cases
> where this is used for FPGAs.
> 
> There are two options how to implement virtio-pci that we've discussed:
> 1. VPCI + IOREQ
> 2. IOREQ only
> 
> There are pros and cons with both. For example with #1, has the benefit that
> we would only have a single PCIe RC (in Xen) and we could emulate a hotplug
> capable expansion port with a standard way to notify when PCI devices plug in.
> Approach #2 has the benefit that there is (almost) no additional complexity
> or code added to Xen, almost everything lives outside.
> IMO, both options have merit and both could co-exist.
> 
> For dynamic xl flows, options #2 already works without modifications to Xen.
> Users need to pass the correct command-line options to QEMU and a device-tree
> fragment with the pci-generic-ecam-host device.
> 
> For static dom0less flows, we can do the same as for xl flows but we have the
> additional problem of domU's PCI bus enumeration racing with QEMU.
> On x86, when domU's access a memory range that has not yet got IOREQ's
> connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> writes ignored. This makes it easy for guests to wait for IOREQ devices to
> pop up since guests will find an empty bus and can initiate a rescan later
> when QEMU attaches. On ARM, we trap on these accesses.
> 
> If we on ARM add support for MMIO background regions with a default read value,
> i.e mmio handlers that have lower priority than IOREQs and that are
> read-const + writes-ignored, we could support the same flow on ARM.
> This may be generally useful for other devices as well (e.g virtio-mmio or
> something else). We could also use this to defer PCI enumeration.
> 
> So the next versions of this series I was thinking to remove the PCI specifics
> and instead add FDT bindings to ARM dom0less enabling setup of user
> configurable (by address-range and read-value) background mmio regions.
> Xen would then support option #2 without any PCI specifics added.
> 
> Thoughts?

We discussed that together last week and I think this is a good approach as it
prevents having some "workaround" code for PCI not following the Virtio spec and
could also fulfil some other use cases by giving a solution to emulate some IO
registers for a guest with a fixed value.

Cheers
Bertrand

> 
> Cheers,
> Edgar
> 
> # References to spec
> 
> PCI express base specification:
> 7.5.1.1.1 Vendor ID Register (Offset 00h)
> The Vendor ID register is HwInit and the value in this register identifies the manufacturer of the Function. In keeping with
> PCI-SIG procedures, valid vendor identifiers must be allocated by the PCI-SIG to ensure uniqueness. Each vendor must
> have at least one Vendor ID. It is recommended that software read the Vendor ID register to determine if a Function is
> present, where a value of FFFFh indicates that no Function is present.
> 
> PCI Firmware Specification:
> 3.5 Device State at Firmware/Operating System Handoff
> Page 34:
> The operating system is required to configure PCI subsystems:
>  During hotplug
>  For devices that take too long to come out of reset
>  PCI-to-PCI bridges that are at levels below what firmware is designed to configure
> 
> Page 36:
> Note: The operating system does not have to walk all buses during boot. The kernel can
> automatically configure devices on request; i.e., an event can cause a scan of I/O on demand.
> 
> FPGA's can be programmed at runtime and appear on the ECAM bus silently.
> An PCI rescan needs to be triggered for the OS to discover the device:
> Intel FPGAs:
> https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html
> 

Re: xen/arm: Virtio-PCI for dom0less on ARM

[Stefano Stabellini]

On Wed, 21 May 2025, Bertrand Marquis wrote:
> Hi Edgar,
> 
> > On 20 May 2025, at 10:04, Edgar E. Iglesias <edgar.iglesias@amd.com> wrote:
> > 
> > Hi all,
> > 
> > Following up on the ARM virtio-pci series I posted a while back ago.
> > 
> > There have been some concerns around the delayed and silent apperance of
> > devices on the ECAM area. The spec is not super clear wether this is OK or
> > not but I'm providing some references to the PCI specs and to some real cases
> > where this is used for FPGAs.
> > 
> > There are two options how to implement virtio-pci that we've discussed:
> > 1. VPCI + IOREQ
> > 2. IOREQ only
> > 
> > There are pros and cons with both. For example with #1, has the benefit that
> > we would only have a single PCIe RC (in Xen) and we could emulate a hotplug
> > capable expansion port with a standard way to notify when PCI devices plug in.
> > Approach #2 has the benefit that there is (almost) no additional complexity
> > or code added to Xen, almost everything lives outside.
> > IMO, both options have merit and both could co-exist.
> > 
> > For dynamic xl flows, options #2 already works without modifications to Xen.
> > Users need to pass the correct command-line options to QEMU and a device-tree
> > fragment with the pci-generic-ecam-host device.
> > 
> > For static dom0less flows, we can do the same as for xl flows but we have the
> > additional problem of domU's PCI bus enumeration racing with QEMU.
> > On x86, when domU's access a memory range that has not yet got IOREQ's
> > connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> > writes ignored. This makes it easy for guests to wait for IOREQ devices to
> > pop up since guests will find an empty bus and can initiate a rescan later
> > when QEMU attaches. On ARM, we trap on these accesses.
> > 
> > If we on ARM add support for MMIO background regions with a default read value,
> > i.e mmio handlers that have lower priority than IOREQs and that are
> > read-const + writes-ignored, we could support the same flow on ARM.
> > This may be generally useful for other devices as well (e.g virtio-mmio or
> > something else). We could also use this to defer PCI enumeration.
> > 
> > So the next versions of this series I was thinking to remove the PCI specifics
> > and instead add FDT bindings to ARM dom0less enabling setup of user
> > configurable (by address-range and read-value) background mmio regions.
> > Xen would then support option #2 without any PCI specifics added.
> > 
> > Thoughts?
> 
> We discussed that together last week and I think this is a good approach as it
> prevents having some "workaround" code for PCI not following the Virtio spec and
> could also fulfil some other use cases by giving a solution to emulate some IO
> registers for a guest with a fixed value.

+1

Re: xen/arm: Virtio-PCI for dom0less on ARM

[Julien Grall]

Hi Edgar,

Thanks for the write-up.

On 20/05/2025 09:04, Edgar E. Iglesias wrote:
> Hi all,
> 
> Following up on the ARM virtio-pci series I posted a while back ago.
> 
> There have been some concerns around the delayed and silent apperance of
> devices on the ECAM area. The spec is not super clear wether this is OK or
> not but I'm providing some references to the PCI specs and to some real cases
> where this is used for FPGAs.
> 
> There are two options how to implement virtio-pci that we've discussed:
> 1. VPCI + IOREQ
> 2. IOREQ only
> 
> There are pros and cons with both. For example with #1, has the benefit that
> we would only have a single PCIe RC (in Xen) and we could emulate a hotplug
> capable expansion port with a standard way to notify when PCI devices plug in.
> Approach #2 has the benefit that there is (almost) no additional complexity
> or code added to Xen, almost everything lives outside.
> IMO, both options have merit and both could co-exist.
 > > For dynamic xl flows, options #2 already works without 
modifications to Xen.
> Users need to pass the correct command-line options to QEMU and a device-tree
> fragment with the pci-generic-ecam-host device.

IIUC, in approach #2, QEMU will emulate the host controller. In Xen, we 
also support multiple IOREQ servers. For instance IOREQ A may emulate a 
GPU device, whereas IOREQ B could emulate a disk. This is usuful in case 
where one may want a separate domain to handle GPUs.

With the approach #2, it sounds like you will end up to have one host 
controller per IOREQ server. The user will also need to know them in 
advance. Is my understanding correct? If so, then it feels like this is 
defeating the purpose of IOREQ.

This is the first reason why I feel approach #1 is more suitable.

> 
> For static dom0less flows, we can do the same as for xl flows but we have the
> additional problem of domU's PCI bus enumeration racing with QEMU.
> On x86, when domU's access a memory range that has not yet got IOREQ's
> connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> writes ignored. This makes it easy for guests to wait for IOREQ devices to
> pop up since guests will find an empty bus and can initiate a rescan later
> when QEMU attaches. On ARM, we trap on these accesses.
> > If we on ARM add support for MMIO background regions with a default 
read value,
> i.e mmio handlers that have lower priority than IOREQs and that are
> read-const + writes-ignored, we could support the same flow on ARM.
> This may be generally useful for other devices as well (e.g virtio-mmio or
> something else). We could also use this to defer PCI enumeration.

Regardless what I wrote above, if we are going down the route of 
returning 0xFFFFFFFF, I would just do it for every IOs rather than the 
one specify in the device-tree. This could still behind a per-domain 
option, but it would at least make simpler to setup the system (AFAIU, 
in your current proposal, we would need to specify the range in multiple 
places).

> 
> So the next versions of this series I was thinking to remove the PCI specifics
> and instead add FDT bindings to ARM dom0less enabling setup of user
> configurable (by address-range and read-value) background mmio regions.
> Xen would then support option #2 without any PCI specifics added.
> 
> Thoughts?
> 
> Cheers,
> Edgar
> 
> # References to spec
> 
> PCI express base specification:
> 7.5.1.1.1 Vendor ID Register (Offset 00h)
> The Vendor ID register is HwInit and the value in this register identifies the manufacturer of the Function. In keeping with
> PCI-SIG procedures, valid vendor identifiers must be allocated by the PCI-SIG to ensure uniqueness. Each vendor must
> have at least one Vendor ID. It is recommended that software read the Vendor ID register to determine if a Function is
> present, where a value of FFFFh indicates that no Function is present.
> 
> PCI Firmware Specification:
> 3.5 Device State at Firmware/Operating System Handoff
> Page 34:
> The operating system is required to configure PCI subsystems:
>  During hotplug
>  For devices that take too long to come out of reset
>  PCI-to-PCI bridges that are at levels below what firmware is designed to configure
> 
> Page 36:
> Note: The operating system does not have to walk all buses during boot. The kernel can
> automatically configure devices on request; i.e., an event can cause a scan of I/O on demand.

I am not sure why you quote this. To me it reads like this is up to the 
OS to decide when to access the PCI bus. As we don't control the OS, 
this doesn't seem a behavior Xen can rely on.

> 
> FPGA's can be programmed at runtime and appear on the ECAM bus silently.
> An PCI rescan needs to be triggered for the OS to discover the device:
> Intel FPGAs:
> https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html

To clarify, you are saying the ECAM bus may be completely empty (e.g. 
everything is reading as ~0) and some part of the ECAM will return a non 
~0 value when the FPGA run.

That said, the FPGA behavior is IMHO slightly different. I would expect 
for FPGA, one would now when the device is present because they would 
have programmed the FPGA. In our case, we are trying to solve a race 
introduce by Xen (not the user itself). So it feels wrong to ask the 
user to "probe in a loop until it works".

This is the other reason why the approach #1 looks more appealing to me.

Cheers,

-- 
Julien Grall

Re: xen/arm: Virtio-PCI for dom0less on ARM

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4022 bytes --]

On Wed, 21 May 2025, Julien Grall wrote:
> > For static dom0less flows, we can do the same as for xl flows but we have
> > the
> > additional problem of domU's PCI bus enumeration racing with QEMU.
> > On x86, when domU's access a memory range that has not yet got IOREQ's
> > connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> > writes ignored. This makes it easy for guests to wait for IOREQ devices to
> > pop up since guests will find an empty bus and can initiate a rescan later
> > when QEMU attaches. On ARM, we trap on these accesses.
> > > If we on ARM add support for MMIO background regions with a default 
> read value,
> > i.e mmio handlers that have lower priority than IOREQs and that are
> > read-const + writes-ignored, we could support the same flow on ARM.
> > This may be generally useful for other devices as well (e.g virtio-mmio or
> > something else). We could also use this to defer PCI enumeration.
> 
> Regardless what I wrote above, if we are going down the route of returning
> 0xFFFFFFFF, I would just do it for every IOs rather than the one specify in
> the device-tree. This could still behind a per-domain option, but it would at
> least make simpler to setup the system (AFAIU, in your current proposal, we
> would need to specify the range in multiple places).
 
This seems like a good idea.


> > So the next versions of this series I was thinking to remove the PCI
> > specifics
> > and instead add FDT bindings to ARM dom0less enabling setup of user
> > configurable (by address-range and read-value) background mmio regions.
> > Xen would then support option #2 without any PCI specifics added.
> > 
> > Thoughts?
> > 
> > Cheers,
> > Edgar
> > 
> > # References to spec
> > 
> > PCI express base specification:
> > 7.5.1.1.1 Vendor ID Register (Offset 00h)
> > The Vendor ID register is HwInit and the value in this register identifies
> > the manufacturer of the Function. In keeping with
> > PCI-SIG procedures, valid vendor identifiers must be allocated by the
> > PCI-SIG to ensure uniqueness. Each vendor must
> > have at least one Vendor ID. It is recommended that software read the Vendor
> > ID register to determine if a Function is
> > present, where a value of FFFFh indicates that no Function is present.
> > 
> > PCI Firmware Specification:
> > 3.5 Device State at Firmware/Operating System Handoff
> > Page 34:
> > The operating system is required to configure PCI subsystems:
> >  During hotplug
> >  For devices that take too long to come out of reset
> >  PCI-to-PCI bridges that are at levels below what firmware is designed to
> > configure
> > 
> > Page 36:
> > Note: The operating system does not have to walk all buses during boot. The
> > kernel can
> > automatically configure devices on request; i.e., an event can cause a scan
> > of I/O on demand.
> 
> I am not sure why you quote this. To me it reads like this is up to the OS to
> decide when to access the PCI bus. As we don't control the OS, this doesn't
> seem a behavior Xen can rely on.
> 
> > 
> > FPGA's can be programmed at runtime and appear on the ECAM bus silently.
> > An PCI rescan needs to be triggered for the OS to discover the device:
> > Intel FPGAs:
> > https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html
> 
> To clarify, you are saying the ECAM bus may be completely empty (e.g.
> everything is reading as ~0) and some part of the ECAM will return a non ~0
> value when the FPGA run.

From what I found from my searches on the topic, the PCIe spec says when
a device is not present or not initialized, the system typically
receives a value of 0xFFFFFFFF on read. Once a device becomes active and
responds to configuration accesses, subsequent reads will return valid
data. The specifications do not require interrupts or hot-plug events
for device detection; polling the configuration space is a compliant
method.

Which actually points to your suggestion above to do this for every IOs.

Re: xen/arm: Virtio-PCI for dom0less on ARM

[Edgar E. Iglesias]

On Wed, May 21, 2025 at 01:22:11PM +0100, Julien Grall wrote:
> Hi Edgar,
> 
> Thanks for the write-up.
> 

Thanks for commenting!


> On 20/05/2025 09:04, Edgar E. Iglesias wrote:
> > Hi all,
> > 
> > Following up on the ARM virtio-pci series I posted a while back ago.
> > 
> > There have been some concerns around the delayed and silent apperance of
> > devices on the ECAM area. The spec is not super clear wether this is OK or
> > not but I'm providing some references to the PCI specs and to some real cases
> > where this is used for FPGAs.
> > 
> > There are two options how to implement virtio-pci that we've discussed:
> > 1. VPCI + IOREQ
> > 2. IOREQ only
> > 
> > There are pros and cons with both. For example with #1, has the benefit that
> > we would only have a single PCIe RC (in Xen) and we could emulate a hotplug
> > capable expansion port with a standard way to notify when PCI devices plug in.
> > Approach #2 has the benefit that there is (almost) no additional complexity
> > or code added to Xen, almost everything lives outside.
> > IMO, both options have merit and both could co-exist.
> > > For dynamic xl flows, options #2 already works without modifications to
> Xen.
> > Users need to pass the correct command-line options to QEMU and a device-tree
> > fragment with the pci-generic-ecam-host device.
> 
> IIUC, in approach #2, QEMU will emulate the host controller. In Xen, we also
> support multiple IOREQ servers. For instance IOREQ A may emulate a GPU
> device, whereas IOREQ B could emulate a disk. This is usuful in case where
> one may want a separate domain to handle GPUs.
> 
> With the approach #2, it sounds like you will end up to have one host
> controller per IOREQ server. The user will also need to know them in
> advance. Is my understanding correct? If so, then it feels like this is
> defeating the purpose of IOREQ.

I don't think that is necessarily the case.

Option #2 would have the controller outside of Xen and an implementation
is free to split it up into multiple processes each with an IOREQ server.
It is also free to do a monolithic implementation or some kind of mix.

Today, QEMU supports 3 ways:
1. A monolithic PCI host bridge + PCI Endpoints all in the same process.
2. A PCI host bridge in one process and distributed PCI Endpoints in separate
   processes (vfio-user).
3. A PCI host bridge with partial virtio-PCI transports in a single process
   + distributed virtio backends (over vhost-user) in separate processes.

If you refer to guest's knowing about PCIe controllers in advance, yes but
I don't see why we couldn't use something like dt-overlays and kernel
modules to enable them at runtime. For PCI endpoints behind the controller,
an expansion port with hotplug can be emulated allowing the hot plugin of
devices at runtime.

With regards to isolation, I don't really see a benefit with option #1.


> 
> This is the first reason why I feel approach #1 is more suitable.
> 
> > 
> > For static dom0less flows, we can do the same as for xl flows but we have the
> > additional problem of domU's PCI bus enumeration racing with QEMU.
> > On x86, when domU's access a memory range that has not yet got IOREQ's
> > connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> > writes ignored. This makes it easy for guests to wait for IOREQ devices to
> > pop up since guests will find an empty bus and can initiate a rescan later
> > when QEMU attaches. On ARM, we trap on these accesses.
> > > If we on ARM add support for MMIO background regions with a default
> read value,
> > i.e mmio handlers that have lower priority than IOREQs and that are
> > read-const + writes-ignored, we could support the same flow on ARM.
> > This may be generally useful for other devices as well (e.g virtio-mmio or
> > something else). We could also use this to defer PCI enumeration.
> 
> Regardless what I wrote above, if we are going down the route of returning
> 0xFFFFFFFF, I would just do it for every IOs rather than the one specify in
> the device-tree. This could still behind a per-domain option, but it would
> at least make simpler to setup the system (AFAIU, in your current proposal,
> we would need to specify the range in multiple places).

Yes, the range would go into Xen's static domain config node and into the
passthrough fragment describing the PCI host bridge. Typically only one
range is needed (ECAM).

In my prototype, I've got this for the static domain config:
domU1 {
  compatible = "xen,domain";
  ...
  mmio-background-regions = < 0xf1 0x0                // Base
                              0x0 0x10000000          // Size
                              0xffffffff 0xffffffff   // Read-value

                              // Additional ranges may follow
                              >;
};

> 
> > 
> > So the next versions of this series I was thinking to remove the PCI specifics
> > and instead add FDT bindings to ARM dom0less enabling setup of user
> > configurable (by address-range and read-value) background mmio regions.
> > Xen would then support option #2 without any PCI specifics added.
> > 
> > Thoughts?
> > 
> > Cheers,
> > Edgar
> > 
> > # References to spec
> > 
> > PCI express base specification:
> > 7.5.1.1.1 Vendor ID Register (Offset 00h)
> > The Vendor ID register is HwInit and the value in this register identifies the manufacturer of the Function. In keeping with
> > PCI-SIG procedures, valid vendor identifiers must be allocated by the PCI-SIG to ensure uniqueness. Each vendor must
> > have at least one Vendor ID. It is recommended that software read the Vendor ID register to determine if a Function is
> > present, where a value of FFFFh indicates that no Function is present.
> > 
> > PCI Firmware Specification:
> > 3.5 Device State at Firmware/Operating System Handoff
> > Page 34:
> > The operating system is required to configure PCI subsystems:
> >  During hotplug
> >  For devices that take too long to come out of reset
> >  PCI-to-PCI bridges that are at levels below what firmware is designed to configure
> > 
> > Page 36:
> > Note: The operating system does not have to walk all buses during boot. The kernel can
> > automatically configure devices on request; i.e., an event can cause a scan of I/O on demand.
> 
> I am not sure why you quote this. To me it reads like this is up to the OS
> to decide when to access the PCI bus. As we don't control the OS, this
> doesn't seem a behavior Xen can rely on.
> 
> > 
> > FPGA's can be programmed at runtime and appear on the ECAM bus silently.
> > An PCI rescan needs to be triggered for the OS to discover the device:
> > Intel FPGAs:
> > https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html
> 
> To clarify, you are saying the ECAM bus may be completely empty (e.g.
> everything is reading as ~0) and some part of the ECAM will return a non ~0
> value when the FPGA run.
> 
> That said, the FPGA behavior is IMHO slightly different. I would expect for
> FPGA, one would now when the device is present because they would have
> programmed the FPGA. In our case, we are trying to solve a race introduce by
> Xen (not the user itself). So it feels wrong to ask the user to "probe in a
> loop until it works".
> 
> This is the other reason why the approach #1 looks more appealing to me.

FPGA's loading has many scenarios, it can be done by the user that also
triggers the PCI rescan or it can by done at boot by another machine or
by another VM. There are ways to get the PCI link up quickly at boot
but they are not always used, introducing a similar problem as the one
with virtio-pci in Xen.

Note that approach #1 does not remove boot dependencies.
For example if a guest wants to boot from a virtio-blk disk, it will get
notified when the disk gets hot-plugged but the kernel may have already
failed to mount the disk. We would need a way to configure guests to
wait for a specific device to appear.

Cheers,
Edgar