Is it possible to support hotplug device to PXB bridge?

Is it possible to support hotplug device to PXB bridge?

[Eric Ren]

Hi Marcel and all,

It fails when I want to hotplug device to PXB bus. Then, I find this
commit that explicitely declares the fact that PXB bus does not support
hotplug device onto it.

7b346c742cd9 ("hw/pxb: declare pxb devices as not hot-pluggable")

Could you please help confirm the possibility to make PXB bus hotpluggable,
and the main work to achieve it if possible?

Yes, pxb-pcie with Q35 do support hotplug, but we use i440 machine a long way,
not  willing to change machine type to have it.

Thanks,
Eric Ren

Re: Is it possible to support hotplug device to PXB bridge?

[Daniel P. Berrangé]

On Fri, Apr 22, 2022 at 03:17:11PM +0800, Eric Ren wrote:
> Hi Marcel and all,
> 
> It fails when I want to hotplug device to PXB bus. Then, I find this
> commit that explicitely declares the fact that PXB bus does not support
> hotplug device onto it.
> 
> 7b346c742cd9 ("hw/pxb: declare pxb devices as not hot-pluggable")
> 
> Could you please help confirm the possibility to make PXB bus hotpluggable,
> and the main work to achieve it if possible?

Instead of trying to hotplug into the PXB, attach a 'pci-bridge' to the
PXB and then you can hotplug into the latter instead.

> Yes, pxb-pcie with Q35 do support hotplug, but we use i440 machine a long way,
> not  willing to change machine type to have it.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Is it possible to support hotplug device to PXB bridge?

[Igor Mammedov]

On Fri, 22 Apr 2022 09:29:20 +0100
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Fri, Apr 22, 2022 at 03:17:11PM +0800, Eric Ren wrote:
> > Hi Marcel and all,
> > 
> > It fails when I want to hotplug device to PXB bus. Then, I find this
> > commit that explicitely declares the fact that PXB bus does not support
> > hotplug device onto it.
> > 
> > 7b346c742cd9 ("hw/pxb: declare pxb devices as not hot-pluggable")
> > 
> > Could you please help confirm the possibility to make PXB bus hotpluggable,
> > and the main work to achieve it if possible?  
> 
> Instead of trying to hotplug into the PXB, attach a 'pci-bridge' to the
> PXB and then you can hotplug into the latter instead.

there is no ACPI based hotplug for devices on pxb if I'm not mistaken, so
you'll likely need to enable shpc on pci-bridge.
 
> > Yes, pxb-pcie with Q35 do support hotplug, but we use i440 machine a long way,
> > not  willing to change machine type to have it.  
> 
> With regards,
> Daniel

Re: Is it possible to support hotplug device to PXB bridge?

[Eric Ren]

Hi Igor, Daniel:

Thanks for the hints!

In the PXB doc (https://github.com/qemu/qemu/blob/master/docs/pci_expander_bridge.txt),
PXB device internally adds a pci bridge, according to the impl brief:

```
The PXB is composed by:
...
- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV)
Created automatically as part of init sequence.
When adding a device to PXB it is attached to the bridge for two reasons:
- Using the bridge will **enable hotplug support**
- All the devices behind the bridge will use bridge's IO/MEM windows compacting
the PCI address space.
```

A PXB device already takes 2 bus nr: 1 for the added root bus via host
bridge, 1 for the pci bridge.
If statically adding 1 more pci bridge, it takes 1 more pci bus...

Anyway, will try to figure out the reason why the doc says "Using the
bridge will enable hotplug support",
but in fact not :-)

The start point seemingly is, to generate correct ACPI SSDT table
having hotplug devices/methods for slots behind the PXB
internal pci bridge.

Regards,
Eric Ren

On Fri, 22 Apr 2022 at 16:57, Igor Mammedov <imammedo@redhat.com> wrote:
>
> On Fri, 22 Apr 2022 09:29:20 +0100
> Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> > On Fri, Apr 22, 2022 at 03:17:11PM +0800, Eric Ren wrote:
> > > Hi Marcel and all,
> > >
> > > It fails when I want to hotplug device to PXB bus. Then, I find this
> > > commit that explicitely declares the fact that PXB bus does not support
> > > hotplug device onto it.
> > >
> > > 7b346c742cd9 ("hw/pxb: declare pxb devices as not hot-pluggable")
> > >
> > > Could you please help confirm the possibility to make PXB bus hotpluggable,
> > > and the main work to achieve it if possible?
> >
> > Instead of trying to hotplug into the PXB, attach a 'pci-bridge' to the
> > PXB and then you can hotplug into the latter instead.
>
> there is no ACPI based hotplug for devices on pxb if I'm not mistaken, so
> you'll likely need to enable shpc on pci-bridge.
>
> > > Yes, pxb-pcie with Q35 do support hotplug, but we use i440 machine a long way,
> > > not  willing to change machine type to have it.
> >
> > With regards,
> > Daniel
>


-- 
- Eric Ren

Re: Is it possible to support hotplug device to PXB bridge?

[Igor Mammedov]

On Fri, 22 Apr 2022 17:33:08 +0800
Eric Ren <renzhengeek@gmail.com> wrote:

> Hi Igor, Daniel:
> 
> Thanks for the hints!
> 
> In the PXB doc (https://github.com/qemu/qemu/blob/master/docs/pci_expander_bridge.txt),
> PXB device internally adds a pci bridge, according to the impl brief:
> 
> ```
> The PXB is composed by:
> ...
> - PCIBridgeDev(TYPE_PCI_BRIDGE_DEV)
> Created automatically as part of init sequence.
> When adding a device to PXB it is attached to the bridge for two reasons:
> - Using the bridge will **enable hotplug support**
> - All the devices behind the bridge will use bridge's IO/MEM windows compacting
> the PCI address space.
> ```
> 
> A PXB device already takes 2 bus nr: 1 for the added root bus via host
> bridge, 1 for the pci bridge.
> If statically adding 1 more pci bridge, it takes 1 more pci bus...
> 
> Anyway, will try to figure out the reason why the doc says "Using the
> bridge will enable hotplug support",
> but in fact not :-)
> 
> The start point seemingly is, to generate correct ACPI SSDT table
> having hotplug devices/methods for slots behind the PXB
> internal pci bridge.
It doesn't look like a trivial thing to impl.

The easiest approach is what Daniel has suggested (provided guest
OS supports SHPC):

 -device pxb,id=bridge1,bus=pci.0,bus_nr=1 \
 -device pci-bridge,id=hpbridge,bus=bridge1,chassis_nr=1,shpc=on

then you can hotplug devices on hpbridge

Alternatively allow shpc on builtin pxb bridge, by adding a property
that could turn it on, as it's explicitly disabled at the moment:

  pxb_dev_realize_common():
     qdev_prop_set_bit(bds, PCI_BRIDGE_DEV_PROP_SHPC, false);

 
> Regards,
> Eric Ren
> 
> On Fri, 22 Apr 2022 at 16:57, Igor Mammedov <imammedo@redhat.com> wrote:
> >
> > On Fri, 22 Apr 2022 09:29:20 +0100
> > Daniel P. Berrangé <berrange@redhat.com> wrote:
> >  
> > > On Fri, Apr 22, 2022 at 03:17:11PM +0800, Eric Ren wrote:  
> > > > Hi Marcel and all,
> > > >
> > > > It fails when I want to hotplug device to PXB bus. Then, I find this
> > > > commit that explicitely declares the fact that PXB bus does not support
> > > > hotplug device onto it.
> > > >
> > > > 7b346c742cd9 ("hw/pxb: declare pxb devices as not hot-pluggable")
> > > >
> > > > Could you please help confirm the possibility to make PXB bus hotpluggable,
> > > > and the main work to achieve it if possible?  
> > >
> > > Instead of trying to hotplug into the PXB, attach a 'pci-bridge' to the
> > > PXB and then you can hotplug into the latter instead.  
> >
> > there is no ACPI based hotplug for devices on pxb if I'm not mistaken, so
> > you'll likely need to enable shpc on pci-bridge.
> >  
> > > > Yes, pxb-pcie with Q35 do support hotplug, but we use i440 machine a long way,
> > > > not  willing to change machine type to have it.  
> > >
> > > With regards,
> > > Daniel  
> >  
> 
>