[PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Jason Chien]

In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
the child bus "dw-pcie" is realized before the parent bus "pcie" which is
the root PCIe bus. Thus, the extended configuration space is not accessible
on "dw-pcie". The issue can be resolved by adding the
PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.

Signed-off-by: Jason Chien <jason.chien@sifive.com>
---
 hw/pci-host/designware.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
index 9e183caa48..388d252ee2 100644
--- a/hw/pci-host/designware.c
+++ b/hw/pci-host/designware.c
@@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState *dev, Error **errp)
                                      &s->pci.io,
                                      0, 4,
                                      TYPE_PCIE_BUS);
+    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
 
     memory_region_init(&s->pci.address_space_root,
                        OBJECT(s),
-- 
2.17.1

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Michael S. Tsirkin]

On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
> bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> the root PCIe bus. Thus, the extended configuration space is not accessible
> on "dw-pcie". The issue can be resolved by adding the
> PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.
> 
> Signed-off-by: Jason Chien <jason.chien@sifive.com>

I think we should fix the order of initialization rather than
hack around it.

> ---
>  hw/pci-host/designware.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
> index 9e183caa48..388d252ee2 100644
> --- a/hw/pci-host/designware.c
> +++ b/hw/pci-host/designware.c
> @@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState *dev, Error **errp)
>                                       &s->pci.io,
>                                       0, 4,
>                                       TYPE_PCIE_BUS);
> +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
>  
>      memory_region_init(&s->pci.address_space_root,
>                         OBJECT(s),
> -- 
> 2.17.1

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Jason Chien]

[-- Attachment #1: Type: text/plain, Size: 2751 bytes --]

As far as I know, the order issue is caused by nested device realization.
In this case, realizing TYPE_DESIGNWARE_PCIE_HOST will also
realize TYPE_DESIGNWARE_PCIE_ROOT(see designware_pcie_host_realize()).
device_set_realized() is the function that realizing a device must go
through, and this function first realizes the device by dc->realize() and
then realizes the device's child bus by qbus_realize(). Whether there is
any child bus of the device may depend on dc->realize(). The realization
flow will be like a recursive call to device_set_realized(). More
precisely, the flow in this case is: qdev_realize() --> ... --> FIRST
device_set_realized() --> FIRST dc->realize() --> ...
--> designware_pcie_host_realize() --> qdev_realize() --> ... --> SECOND
device_set_realized() --> SECOND dc->realize() --> ...
--> designware_pcie_root_realize() --> ...--> back to the SECOND
device_set_realized() --> SECOND qbus_realize() the CHILD bus "dw-pcie" -->
... --> back to the FIRST device_set_realized() --> FIRST qbus_realize()
the PARENT bus "pcie".

I also found this patch
<https://lists.gnu.org/archive/html/qemu-devel/2021-11/msg02162.html> that
solves the same bus issue.

Do you have any suggestions on the order of realization? Thanks!

On Thu, Aug 10, 2023 at 5:24 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a
> non-root
> > bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> > the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> > the root PCIe bus. Thus, the extended configuration space is not
> accessible
> > on "dw-pcie". The issue can be resolved by adding the
> > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is
> realized.
> >
> > Signed-off-by: Jason Chien <jason.chien@sifive.com>
>
> I think we should fix the order of initialization rather than
> hack around it.
>
> > ---
> >  hw/pci-host/designware.c | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
> > index 9e183caa48..388d252ee2 100644
> > --- a/hw/pci-host/designware.c
> > +++ b/hw/pci-host/designware.c
> > @@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState
> *dev, Error **errp)
> >                                       &s->pci.io,
> >                                       0, 4,
> >                                       TYPE_PCIE_BUS);
> > +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
> >
> >      memory_region_init(&s->pci.address_space_root,
> >                         OBJECT(s),
> > --
> > 2.17.1
>
>

[-- Attachment #2: Type: text/html, Size: 3533 bytes --]

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Michael S. Tsirkin]

On Fri, Aug 11, 2023 at 01:22:08AM +0800, Jason Chien wrote:
> As far as I know, the order issue is caused by nested device realization. In
> this case, realizing TYPE_DESIGNWARE_PCIE_HOST will also
> realize TYPE_DESIGNWARE_PCIE_ROOT(see designware_pcie_host_realize()).
> device_set_realized() is the function that realizing a device must go through,
> and this function first realizes the device by dc->realize() and then realizes
> the device's child bus by qbus_realize(). Whether there is any child bus of the
> device may depend on dc->realize(). The realization flow will be like a
> recursive call to device_set_realized(). More precisely, the flow in this case
> is: qdev_realize() --> ... --> FIRST device_set_realized() --> FIRST dc->
> realize() --> ... --> designware_pcie_host_realize() --> qdev_realize() --> ...
> --> SECOND device_set_realized() --> SECOND dc->realize() --> ... -->
>  designware_pcie_root_realize() --> ...--> back to the SECOND
> device_set_realized() --> SECOND qbus_realize() the CHILD bus "dw-pcie" --> ...
> --> back to the FIRST device_set_realized() --> FIRST qbus_realize() the PARENT
> bus "pcie".
> 
> I also found this patch that solves the same bus issue.

Which patch?

> Do you have any suggestions on the order of realization? Thanks!


I see. It's not easy to fix. Worth thinking about but I guess your
patch is ok for now.

> On Thu, Aug 10, 2023 at 5:24 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> 
>     On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
>     > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a
>     non-root
>     > bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
>     > the child bus "dw-pcie" is realized before the parent bus "pcie" which is
>     > the root PCIe bus. Thus, the extended configuration space is not
>     accessible
>     > on "dw-pcie". The issue can be resolved by adding the
>     > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is
>     realized.
>     >
>     > Signed-off-by: Jason Chien <jason.chien@sifive.com>
> 
>     I think we should fix the order of initialization rather than
>     hack around it.
> 
>     > ---
>     >  hw/pci-host/designware.c | 1 +
>     >  1 file changed, 1 insertion(+)
>     >
>     > diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
>     > index 9e183caa48..388d252ee2 100644
>     > --- a/hw/pci-host/designware.c
>     > +++ b/hw/pci-host/designware.c
>     > @@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState
>     *dev, Error **errp)
>     >                                       &s->pci.io,
>     >                                       0, 4,
>     >                                       TYPE_PCIE_BUS);
>     > +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
>     > 
>     >      memory_region_init(&s->pci.address_space_root,
>     >                         OBJECT(s),
>     > --
>     > 2.17.1
> 
> 

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Jason Chien]

[-- Attachment #1: Type: text/plain, Size: 3430 bytes --]

the patch
<https://lists.gnu.org/archive/html/qemu-devel/2021-11/msg02162.html> link:
https://lists.gnu.org/archive/html/qemu-devel/2021-11/msg02162.html

On Fri, Aug 11, 2023 at 1:44 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> On Fri, Aug 11, 2023 at 01:22:08AM +0800, Jason Chien wrote:
> > As far as I know, the order issue is caused by nested device
> realization. In
> > this case, realizing TYPE_DESIGNWARE_PCIE_HOST will also
> > realize TYPE_DESIGNWARE_PCIE_ROOT(see designware_pcie_host_realize()).
> > device_set_realized() is the function that realizing a device must go
> through,
> > and this function first realizes the device by dc->realize() and then
> realizes
> > the device's child bus by qbus_realize(). Whether there is any child bus
> of the
> > device may depend on dc->realize(). The realization flow will be like a
> > recursive call to device_set_realized(). More precisely, the flow in
> this case
> > is: qdev_realize() --> ... --> FIRST device_set_realized() --> FIRST dc->
> > realize() --> ... --> designware_pcie_host_realize() --> qdev_realize()
> --> ...
> > --> SECOND device_set_realized() --> SECOND dc->realize() --> ... -->
> >  designware_pcie_root_realize() --> ...--> back to the SECOND
> > device_set_realized() --> SECOND qbus_realize() the CHILD bus "dw-pcie"
> --> ...
> > --> back to the FIRST device_set_realized() --> FIRST qbus_realize() the
> PARENT
> > bus "pcie".
> >
> > I also found this patch that solves the same bus issue.
>
> Which patch?
>
> > Do you have any suggestions on the order of realization? Thanks!
>
>
> I see. It's not easy to fix. Worth thinking about but I guess your
> patch is ok for now.
>
> > On Thu, Aug 10, 2023 at 5:24 AM Michael S. Tsirkin <mst@redhat.com>
> wrote:
> >
> >     On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> >     > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a
> >     non-root
> >     > bus is realized as a PCIe bus if its parent bus is a PCIe bus.
> However,
> >     > the child bus "dw-pcie" is realized before the parent bus "pcie"
> which is
> >     > the root PCIe bus. Thus, the extended configuration space is not
> >     accessible
> >     > on "dw-pcie". The issue can be resolved by adding the
> >     > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is
> >     realized.
> >     >
> >     > Signed-off-by: Jason Chien <jason.chien@sifive.com>
> >
> >     I think we should fix the order of initialization rather than
> >     hack around it.
> >
> >     > ---
> >     >  hw/pci-host/designware.c | 1 +
> >     >  1 file changed, 1 insertion(+)
> >     >
> >     > diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
> >     > index 9e183caa48..388d252ee2 100644
> >     > --- a/hw/pci-host/designware.c
> >     > +++ b/hw/pci-host/designware.c
> >     > @@ -694,6 +694,7 @@ static void
> designware_pcie_host_realize(DeviceState
> >     *dev, Error **errp)
> >     >                                       &s->pci.io,
> >     >                                       0, 4,
> >     >                                       TYPE_PCIE_BUS);
> >     > +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
> >     >
> >     >      memory_region_init(&s->pci.address_space_root,
> >     >                         OBJECT(s),
> >     > --
> >     > 2.17.1
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 4689 bytes --]

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Michael S. Tsirkin]

On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
> bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> the root PCIe bus. Thus, the extended configuration space is not accessible
> on "dw-pcie". The issue can be resolved by adding the
> PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.
> 
> Signed-off-by: Jason Chien <jason.chien@sifive.com>


Acked-by: Michael S. Tsirkin <mst@redhat.com>

I'm not planning another pull before release, hopefully
another maintainer can pick it up? Peter?

> ---
>  hw/pci-host/designware.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
> index 9e183caa48..388d252ee2 100644
> --- a/hw/pci-host/designware.c
> +++ b/hw/pci-host/designware.c
> @@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState *dev, Error **errp)
>                                       &s->pci.io,
>                                       0, 4,
>                                       TYPE_PCIE_BUS);
> +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
>  
>      memory_region_init(&s->pci.address_space_root,
>                         OBJECT(s),
> -- 
> 2.17.1

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Peter Maydell]

On Thu, 10 Aug 2023 at 18:51, Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
> > bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> > the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> > the root PCIe bus. Thus, the extended configuration space is not accessible
> > on "dw-pcie". The issue can be resolved by adding the
> > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.
> >
> > Signed-off-by: Jason Chien <jason.chien@sifive.com>
>
>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
>
> I'm not planning another pull before release, hopefully
> another maintainer can pick it up? Peter?

At the moment I don't have anything intended for 8.1 either,
so whoever of us does it it would be a 1-patch pullreq...

-- PMM

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Peter Maydell]

On Fri, 11 Aug 2023 at 10:55, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Thu, 10 Aug 2023 at 18:51, Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> > > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
> > > bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> > > the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> > > the root PCIe bus. Thus, the extended configuration space is not accessible
> > > on "dw-pcie". The issue can be resolved by adding the
> > > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.
> > >
> > > Signed-off-by: Jason Chien <jason.chien@sifive.com>
> >
> >
> > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> >
> > I'm not planning another pull before release, hopefully
> > another maintainer can pick it up? Peter?
>
> At the moment I don't have anything intended for 8.1 either,
> so whoever of us does it it would be a 1-patch pullreq...

Also, at this stage in the release cycle it's always worth
asking: is this a regression, or was this bug also in 8.0?

thanks
-- PMM

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Jason Chien]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

This bug was also in 8.0.

Jason

On Fri, Aug 11, 2023 at 10:07 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 11 Aug 2023 at 10:55, Peter Maydell <peter.maydell@linaro.org>
> wrote:
> >
> > On Thu, 10 Aug 2023 at 18:51, Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Aug 09, 2023 at 10:22:50AM +0000, Jason Chien wrote:
> > > > In pcie_bus_realize(), a root bus is realized as a PCIe bus and a
> non-root
> > > > bus is realized as a PCIe bus if its parent bus is a PCIe bus.
> However,
> > > > the child bus "dw-pcie" is realized before the parent bus "pcie"
> which is
> > > > the root PCIe bus. Thus, the extended configuration space is not
> accessible
> > > > on "dw-pcie". The issue can be resolved by adding the
> > > > PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is
> realized.
> > > >
> > > > Signed-off-by: Jason Chien <jason.chien@sifive.com>
> > >
> > >
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > > I'm not planning another pull before release, hopefully
> > > another maintainer can pick it up? Peter?
> >
> > At the moment I don't have anything intended for 8.1 either,
> > so whoever of us does it it would be a 1-patch pullreq...
>
> Also, at this stage in the release cycle it's always worth
> asking: is this a regression, or was this bug also in 8.0?
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 2172 bytes --]

Re: [PATCH] hw/pci-host: Allow extended config space access for Designware PCIe host

[Frank Chang]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

Reviewed-by: Frank Chang <frank.chang@sifive.com>

On Wed, Aug 9, 2023 at 6:23 PM Jason Chien <jason.chien@sifive.com> wrote:

> In pcie_bus_realize(), a root bus is realized as a PCIe bus and a non-root
> bus is realized as a PCIe bus if its parent bus is a PCIe bus. However,
> the child bus "dw-pcie" is realized before the parent bus "pcie" which is
> the root PCIe bus. Thus, the extended configuration space is not accessible
> on "dw-pcie". The issue can be resolved by adding the
> PCI_BUS_EXTENDED_CONFIG_SPACE flag to "pcie" before "dw-pcie" is realized.
>
> Signed-off-by: Jason Chien <jason.chien@sifive.com>
> ---
>  hw/pci-host/designware.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
> index 9e183caa48..388d252ee2 100644
> --- a/hw/pci-host/designware.c
> +++ b/hw/pci-host/designware.c
> @@ -694,6 +694,7 @@ static void designware_pcie_host_realize(DeviceState
> *dev, Error **errp)
>                                       &s->pci.io,
>                                       0, 4,
>                                       TYPE_PCIE_BUS);
> +    pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
>
>      memory_region_init(&s->pci.address_space_root,
>                         OBJECT(s),
> --
> 2.17.1
>
>
>

[-- Attachment #2: Type: text/html, Size: 1982 bytes --]