[PATCH v2 00/19] ppc/pnv: Add support for user created PHB3/PHB4 devices

[PATCH v2 00/19] ppc/pnv: Add support for user created PHB3/PHB4 devices

[Cédric Le Goater]

Hello,

On the POWER8 processor, powernv8 machine, PHB3 devices can simply be
created with :

   -device pnv-phb3,chip-id=0,index=1 

with a maximum of 3 PHB3s per chip, each PHB3 adding a new PCIe bus.

On the POWER9 processor, powernv9 machine, the logic is different. The
the chip comes with 3 PHB4 PECs (PCI Express Controller) and each PEC
can have several PHBs :

  * PEC0 provides 1 PHB  (PHB0)
  * PEC1 provides 2 PHBs (PHB1 and PHB2)
  * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)

The PEC devices can be created with :

   -device pnv-phb4-pec,chip-id=0,index=1

And the number of added PHB4 devices depends on the PEC index. Each
PHB4 adds a new PCIe bus.

The following changes are mostly cleanups and improvements of the
PHB3/4 realize routines to enable support. One important change is
related to the way the powernv machine populates the device tree. It
depends on the object hierarchy and it is necessary to reparent user
created devices to the chip they belong to (see PATCH 5). PHB3 is a
little more sophisticated because of its SysBusDevice nature (see
PATCH 6).

It would be preferable for libvirt and user to add one PHB4 (one PCIe
bus) at a time but that's another step. The plan is to merge real soon
the first patches which are required cleanups of the models and give
some more time for the last ones.

Thanks,

C.

Changes in v2:

 - Addressed Frederic's comments 
 - Reworked patchset order to allow pluggable devices at the end.
 
Cédric Le Goater (19):
  ppc/pnv: Change the maximum of PHB3 devices for Power8NVL
  ppc/pnv: Introduce a "chip" property under the PHB3 model
  ppc/pnv: Use the chip class to check the index of PHB3 devices
  ppc/pnv: Drop the "num-phbs" property
  ppc/pnv: Move mapping of the PHB3 CQ regions under pnv_pbcq_realize()
  ppc/pnv: Use QOM hierarchy to scan PHB3 devices
  ppc/pnv: Introduce a num_pecs class attribute for PHB4 PEC devices
  ppc/pnv: Introduce version and device_id class atributes for PHB4
    devices
  ppc/pnv: Introduce a "chip" property under the PHB4 model
  ppc/pnv: Introduce a num_stack class attribute
  ppc/pnv: Compute the PHB index from the PHB4 PEC model
  ppc/pnv: Remove "system-memory" property from PHB4 PEC
  ppc/pnv: Move realize of PEC stacks under the PEC model
  ppc/pnv: Use QOM hierarchy to scan PEC PHB4 devices
  ppc/pnv: Introduce support for user created PHB3 devices
  ppc/pnv: Reparent user created PHB3 devices to the PnvChip
  ppc/pnv: Complete user created PHB3 devices
  ppc/pnv: Introduce support for user created PHB4 devices
  ppc/pnv: Move num_phbs under Pnv8Chip

 include/hw/pci-host/pnv_phb3.h |   3 +
 include/hw/pci-host/pnv_phb4.h |   5 +
 include/hw/ppc/pnv.h           |   7 +-
 hw/pci-host/pnv_phb3.c         |  29 ++++-
 hw/pci-host/pnv_phb3_pbcq.c    |  11 ++
 hw/pci-host/pnv_phb4_pec.c     |  94 ++++++++++++--
 hw/ppc/pnv.c                   | 218 ++++++++++++++++++---------------
 7 files changed, 256 insertions(+), 111 deletions(-)

-- 
2.31.1

[PATCH v2 01/19] ppc/pnv: Change the maximum of PHB3 devices for Power8NVL

[Cédric Le Goater]

The POWER8 processors with a NVLink logic unit have 4 PHB3 devices per
chip.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/ppc/pnv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 44ae41a9cb6b..6359bce549ca 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1309,7 +1309,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
 
     k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
     k->cores_mask = POWER8_CORE_MASK;
-    k->num_phbs = 3;
+    k->num_phbs = 4;
     k->core_pir = pnv_chip_core_pir_p8;
     k->intc_create = pnv_chip_power8_intc_create;
     k->intc_reset = pnv_chip_power8_intc_reset;
-- 
2.31.1

Re: [PATCH v2 01/19] ppc/pnv: Change the maximum of PHB3 devices for Power8NVL

[Daniel Henrique Barboza]

On 12/13/21 10:28, Cédric Le Goater wrote:
> The POWER8 processors with a NVLink logic unit have 4 PHB3 devices per
> chip.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>

>   hw/ppc/pnv.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 44ae41a9cb6b..6359bce549ca 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -1309,7 +1309,7 @@ static void pnv_chip_power8nvl_class_init(ObjectClass *klass, void *data)
>   
>       k->chip_cfam_id = 0x120d304980000000ull;  /* P8 Naples DD1.0 */
>       k->cores_mask = POWER8_CORE_MASK;
> -    k->num_phbs = 3;
> +    k->num_phbs = 4;
>       k->core_pir = pnv_chip_core_pir_p8;
>       k->intc_create = pnv_chip_power8_intc_create;
>       k->intc_reset = pnv_chip_power8_intc_reset;
> 

[PATCH v2 02/19] ppc/pnv: Introduce a "chip" property under PHB3

[Cédric Le Goater]

This change will help us move the mapping of XSCOM regions under the
PHB3 realize routine, which will be necessary for user created PHB3
devices.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/pci-host/pnv_phb3.h | 3 +++
 hw/pci-host/pnv_phb3.c         | 1 +
 hw/ppc/pnv.c                   | 2 ++
 3 files changed, 6 insertions(+)

diff --git a/include/hw/pci-host/pnv_phb3.h b/include/hw/pci-host/pnv_phb3.h
index e2a2e3624532..e9c13e6bd821 100644
--- a/include/hw/pci-host/pnv_phb3.h
+++ b/include/hw/pci-host/pnv_phb3.h
@@ -16,6 +16,7 @@
 #include "qom/object.h"
 
 typedef struct PnvPHB3 PnvPHB3;
+typedef struct PnvChip PnvChip;
 
 /*
  * PHB3 XICS Source for MSIs
@@ -157,6 +158,8 @@ struct PnvPHB3 {
     PnvPHB3RootPort root;
 
     QLIST_HEAD(, PnvPhb3DMASpace) dma_spaces;
+
+    PnvChip *chip;
 };
 
 uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size);
diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index a7f96850055a..3aa42ef9d4b9 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -1092,6 +1092,7 @@ static const char *pnv_phb3_root_bus_path(PCIHostState *host_bridge,
 static Property pnv_phb3_properties[] = {
         DEFINE_PROP_UINT32("index", PnvPHB3, phb_id, 0),
         DEFINE_PROP_UINT32("chip-id", PnvPHB3, chip_id, 0),
+        DEFINE_PROP_LINK("chip", PnvPHB3, chip, TYPE_PNV_CHIP, PnvChip *),
         DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 6359bce549ca..74c25c1c5d98 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1231,6 +1231,8 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
         object_property_set_int(OBJECT(phb), "index", i, &error_fatal);
         object_property_set_int(OBJECT(phb), "chip-id", chip->chip_id,
                                 &error_fatal);
+        object_property_set_link(OBJECT(phb), "chip", OBJECT(chip),
+                                 &error_fatal);
         if (!sysbus_realize(SYS_BUS_DEVICE(phb), errp)) {
             return;
         }
-- 
2.31.1

[PATCH v2 03/19] ppc/pnv: Use the chip class to check the index of PHB3 devices

[Cédric Le Goater]

The maximum number of PHB3 devices per chip can be different depending
on the POWER8 processor model.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb3.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index 3aa42ef9d4b9..9c4451ca0d1c 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -993,7 +993,7 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
     PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
     int i;
 
-    if (phb->phb_id >= PNV8_CHIP_PHB3_MAX) {
+    if (phb->phb_id >= PNV_CHIP_GET_CLASS(phb->chip)->num_phbs) {
         error_setg(errp, "invalid PHB index: %d", phb->phb_id);
         return;
     }
-- 
2.31.1

Re: [PATCH v2 03/19] ppc/pnv: Use the chip class to check the index of PHB3 devices

[Daniel Henrique Barboza]

On 12/13/21 10:28, Cédric Le Goater wrote:
> The maximum number of PHB3 devices per chip can be different depending
> on the POWER8 processor model.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>

>   hw/pci-host/pnv_phb3.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
> index 3aa42ef9d4b9..9c4451ca0d1c 100644
> --- a/hw/pci-host/pnv_phb3.c
> +++ b/hw/pci-host/pnv_phb3.c
> @@ -993,7 +993,7 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
>       PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
>       int i;
>   
> -    if (phb->phb_id >= PNV8_CHIP_PHB3_MAX) {
> +    if (phb->phb_id >= PNV_CHIP_GET_CLASS(phb->chip)->num_phbs) {
>           error_setg(errp, "invalid PHB index: %d", phb->phb_id);
>           return;
>       }
> 

[PATCH v2 04/19] ppc/pnv: Drop the "num-phbs" property

[Cédric Le Goater]

It is never used.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/ppc/pnv.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 74c25c1c5d98..0413d221b311 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1771,7 +1771,6 @@ static Property pnv_chip_properties[] = {
     DEFINE_PROP_UINT32("nr-cores", PnvChip, nr_cores, 1),
     DEFINE_PROP_UINT64("cores-mask", PnvChip, cores_mask, 0x0),
     DEFINE_PROP_UINT32("nr-threads", PnvChip, nr_threads, 1),
-    DEFINE_PROP_UINT32("num-phbs", PnvChip, num_phbs, 0),
     DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.31.1

[PATCH v2 05/19] ppc/pnv: Move mapping of the PHB3 CQ regions under pnv_pbcq_realize()

[Cédric Le Goater]

This change will help us providing support for user created PHB3
devices.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb3_pbcq.c | 11 +++++++++++
 hw/ppc/pnv.c                | 12 ------------
 2 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/hw/pci-host/pnv_phb3_pbcq.c b/hw/pci-host/pnv_phb3_pbcq.c
index a0526aa1eca3..c7426cd27a20 100644
--- a/hw/pci-host/pnv_phb3_pbcq.c
+++ b/hw/pci-host/pnv_phb3_pbcq.c
@@ -284,6 +284,17 @@ static void pnv_pbcq_realize(DeviceState *dev, Error **errp)
     pnv_xscom_region_init(&pbcq->xscom_spci_regs, OBJECT(dev),
                           &pnv_pbcq_spci_xscom_ops, pbcq, name,
                           PNV_XSCOM_PBCQ_SPCI_SIZE);
+
+    /* Populate the XSCOM address space. */
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
+                            &pbcq->xscom_nest_regs);
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
+                            &pbcq->xscom_pci_regs);
+    pnv_xscom_add_subregion(phb->chip,
+                            PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
+                            &pbcq->xscom_spci_regs);
 }
 
 static int pnv_pbcq_dt_xscom(PnvXScomInterface *dev, void *fdt,
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 0413d221b311..65196a2a5d00 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1226,7 +1226,6 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
     /* PHB3 controllers */
     for (i = 0; i < chip->num_phbs; i++) {
         PnvPHB3 *phb = &chip8->phbs[i];
-        PnvPBCQState *pbcq = &phb->pbcq;
 
         object_property_set_int(OBJECT(phb), "index", i, &error_fatal);
         object_property_set_int(OBJECT(phb), "chip-id", chip->chip_id,
@@ -1236,17 +1235,6 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
         if (!sysbus_realize(SYS_BUS_DEVICE(phb), errp)) {
             return;
         }
-
-        /* Populate the XSCOM address space. */
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_NEST_BASE + 0x400 * phb->phb_id,
-                                &pbcq->xscom_nest_regs);
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_PCI_BASE + 0x400 * phb->phb_id,
-                                &pbcq->xscom_pci_regs);
-        pnv_xscom_add_subregion(chip,
-                                PNV_XSCOM_PBCQ_SPCI_BASE + 0x040 * phb->phb_id,
-                                &pbcq->xscom_spci_regs);
     }
 }
 
-- 
2.31.1

[PATCH v2 06/19] ppc/pnv: Use QOM hierarchy to scan PHB3 devices

[Cédric Le Goater]

When -nodefaults is supported for PHB3 devices, the phbs array under
the chip will be empty. This will break the XICSFabric handlers, and
all interrupt delivery, and the 'info pic' HMP command.

Do a QOM loop on the chip children and look for PHB3 devices instead.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/ppc/pnv.c | 72 +++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 18 deletions(-)

diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 65196a2a5d00..cbc3f8ed62f7 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -638,16 +638,25 @@ static ISABus *pnv_isa_create(PnvChip *chip, Error **errp)
     return PNV_CHIP_GET_CLASS(chip)->isa_create(chip, errp);
 }
 
+static int pnv_chip_power8_pic_print_info_child(Object *child, void *opaque)
+{
+    Monitor *mon = opaque;
+    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
+
+    if (phb3) {
+        pnv_phb3_msi_pic_print_info(&phb3->msis, mon);
+        ics_pic_print_info(&phb3->lsis, mon);
+    }
+    return 0;
+}
+
 static void pnv_chip_power8_pic_print_info(PnvChip *chip, Monitor *mon)
 {
     Pnv8Chip *chip8 = PNV8_CHIP(chip);
-    int i;
 
     ics_pic_print_info(&chip8->psi.ics, mon);
-    for (i = 0; i < chip->num_phbs; i++) {
-        pnv_phb3_msi_pic_print_info(&chip8->phbs[i].msis, mon);
-        ics_pic_print_info(&chip8->phbs[i].lsis, mon);
-    }
+    object_child_foreach(OBJECT(chip),
+                         pnv_chip_power8_pic_print_info_child, mon);
 }
 
 static void pnv_chip_power9_pic_print_info(PnvChip *chip, Monitor *mon)
@@ -1789,10 +1798,32 @@ PowerPCCPU *pnv_chip_find_cpu(PnvChip *chip, uint32_t pir)
     return NULL;
 }
 
+typedef struct ForeachPhb3Args {
+    int irq;
+    ICSState *ics;
+} ForeachPhb3Args;
+
+static int pnv_ics_get_child(Object *child, void *opaque)
+{
+    ForeachPhb3Args *args = opaque;
+    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
+
+    if (phb3) {
+        if (ics_valid_irq(&phb3->lsis, args->irq)) {
+            args->ics = &phb3->lsis;
+        }
+        if (ics_valid_irq(ICS(&phb3->msis), args->irq)) {
+            args->ics = ICS(&phb3->msis);
+        }
+    }
+    return args->ics ? 1 : 0;
+}
+
 static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
 {
     PnvMachineState *pnv = PNV_MACHINE(xi);
-    int i, j;
+    ForeachPhb3Args args = { irq, NULL };
+    int i;
 
     for (i = 0; i < pnv->num_chips; i++) {
         PnvChip *chip = pnv->chips[i];
@@ -1801,32 +1832,37 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
         if (ics_valid_irq(&chip8->psi.ics, irq)) {
             return &chip8->psi.ics;
         }
-        for (j = 0; j < chip->num_phbs; j++) {
-            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
-                return &chip8->phbs[j].lsis;
-            }
-            if (ics_valid_irq(ICS(&chip8->phbs[j].msis), irq)) {
-                return ICS(&chip8->phbs[j].msis);
-            }
+
+        object_child_foreach(OBJECT(chip), pnv_ics_get_child, &args);
+        if (args.ics) {
+            return args.ics;
         }
     }
     return NULL;
 }
 
+static int pnv_ics_resend_child(Object *child, void *opaque)
+{
+    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
+
+    if (phb3) {
+        ics_resend(&phb3->lsis);
+        ics_resend(ICS(&phb3->msis));
+    }
+    return 0;
+}
+
 static void pnv_ics_resend(XICSFabric *xi)
 {
     PnvMachineState *pnv = PNV_MACHINE(xi);
-    int i, j;
+    int i;
 
     for (i = 0; i < pnv->num_chips; i++) {
         PnvChip *chip = pnv->chips[i];
         Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
 
         ics_resend(&chip8->psi.ics);
-        for (j = 0; j < chip->num_phbs; j++) {
-            ics_resend(&chip8->phbs[j].lsis);
-            ics_resend(ICS(&chip8->phbs[j].msis));
-        }
+        object_child_foreach(OBJECT(chip), pnv_ics_resend_child, NULL);
     }
 }
 
-- 
2.31.1

Re: [PATCH v2 06/19] ppc/pnv: Use QOM hierarchy to scan PHB3 devices

[Daniel Henrique Barboza]

On 12/13/21 10:28, Cédric Le Goater wrote:
> When -nodefaults is supported for PHB3 devices, the phbs array under
> the chip will be empty. This will break the XICSFabric handlers, and
> all interrupt delivery, and the 'info pic' HMP command.
> 
> Do a QOM loop on the chip children and look for PHB3 devices instead.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>

>   hw/ppc/pnv.c | 72 +++++++++++++++++++++++++++++++++++++++-------------
>   1 file changed, 54 insertions(+), 18 deletions(-)
> 
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 65196a2a5d00..cbc3f8ed62f7 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -638,16 +638,25 @@ static ISABus *pnv_isa_create(PnvChip *chip, Error **errp)
>       return PNV_CHIP_GET_CLASS(chip)->isa_create(chip, errp);
>   }
>   
> +static int pnv_chip_power8_pic_print_info_child(Object *child, void *opaque)
> +{
> +    Monitor *mon = opaque;
> +    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
> +
> +    if (phb3) {
> +        pnv_phb3_msi_pic_print_info(&phb3->msis, mon);
> +        ics_pic_print_info(&phb3->lsis, mon);
> +    }
> +    return 0;
> +}
> +
>   static void pnv_chip_power8_pic_print_info(PnvChip *chip, Monitor *mon)
>   {
>       Pnv8Chip *chip8 = PNV8_CHIP(chip);
> -    int i;
>   
>       ics_pic_print_info(&chip8->psi.ics, mon);
> -    for (i = 0; i < chip->num_phbs; i++) {
> -        pnv_phb3_msi_pic_print_info(&chip8->phbs[i].msis, mon);
> -        ics_pic_print_info(&chip8->phbs[i].lsis, mon);
> -    }
> +    object_child_foreach(OBJECT(chip),
> +                         pnv_chip_power8_pic_print_info_child, mon);
>   }
>   
>   static void pnv_chip_power9_pic_print_info(PnvChip *chip, Monitor *mon)
> @@ -1789,10 +1798,32 @@ PowerPCCPU *pnv_chip_find_cpu(PnvChip *chip, uint32_t pir)
>       return NULL;
>   }
>   
> +typedef struct ForeachPhb3Args {
> +    int irq;
> +    ICSState *ics;
> +} ForeachPhb3Args;
> +
> +static int pnv_ics_get_child(Object *child, void *opaque)
> +{
> +    ForeachPhb3Args *args = opaque;
> +    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
> +
> +    if (phb3) {
> +        if (ics_valid_irq(&phb3->lsis, args->irq)) {
> +            args->ics = &phb3->lsis;
> +        }
> +        if (ics_valid_irq(ICS(&phb3->msis), args->irq)) {
> +            args->ics = ICS(&phb3->msis);
> +        }
> +    }
> +    return args->ics ? 1 : 0;
> +}
> +
>   static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>   {
>       PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i, j;
> +    ForeachPhb3Args args = { irq, NULL };
> +    int i;
>   
>       for (i = 0; i < pnv->num_chips; i++) {
>           PnvChip *chip = pnv->chips[i];
> @@ -1801,32 +1832,37 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
>           if (ics_valid_irq(&chip8->psi.ics, irq)) {
>               return &chip8->psi.ics;
>           }
> -        for (j = 0; j < chip->num_phbs; j++) {
> -            if (ics_valid_irq(&chip8->phbs[j].lsis, irq)) {
> -                return &chip8->phbs[j].lsis;
> -            }
> -            if (ics_valid_irq(ICS(&chip8->phbs[j].msis), irq)) {
> -                return ICS(&chip8->phbs[j].msis);
> -            }
> +
> +        object_child_foreach(OBJECT(chip), pnv_ics_get_child, &args);
> +        if (args.ics) {
> +            return args.ics;
>           }
>       }
>       return NULL;
>   }
>   
> +static int pnv_ics_resend_child(Object *child, void *opaque)
> +{
> +    PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
> +
> +    if (phb3) {
> +        ics_resend(&phb3->lsis);
> +        ics_resend(ICS(&phb3->msis));
> +    }
> +    return 0;
> +}
> +
>   static void pnv_ics_resend(XICSFabric *xi)
>   {
>       PnvMachineState *pnv = PNV_MACHINE(xi);
> -    int i, j;
> +    int i;
>   
>       for (i = 0; i < pnv->num_chips; i++) {
>           PnvChip *chip = pnv->chips[i];
>           Pnv8Chip *chip8 = PNV8_CHIP(pnv->chips[i]);
>   
>           ics_resend(&chip8->psi.ics);
> -        for (j = 0; j < chip->num_phbs; j++) {
> -            ics_resend(&chip8->phbs[j].lsis);
> -            ics_resend(ICS(&chip8->phbs[j].msis));
> -        }
> +        object_child_foreach(OBJECT(chip), pnv_ics_resend_child, NULL);
>       }
>   }
>   
> 

[PATCH v2 07/19] ppc/pnv: Introduce a num_pecs class attribute for PHB4 PEC devices

[Cédric Le Goater]

POWER9 processor comes with 3 PHB4 PEC (PCI Express Controller) and
each PEC can have several PHBs :

  * PEC0 provides 1 PHB  (PHB0)
  * PEC1 provides 2 PHBs (PHB1 and PHB2)
  * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)

A num_pecs class attribute represents better the logic units of the
POWER9 chip. Use that instead of num_phbs which fits POWER8 chips.
This will ease adding support for user created devices.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/pnv.h |  2 ++
 hw/ppc/pnv.c         | 19 ++++++++-----------
 2 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
index aa08d79d24de..c781525277db 100644
--- a/include/hw/ppc/pnv.h
+++ b/include/hw/ppc/pnv.h
@@ -53,6 +53,7 @@ struct PnvChip {
     PnvCore      **cores;
 
     uint32_t     num_phbs;
+    uint32_t     num_pecs;
 
     MemoryRegion xscom_mmio;
     MemoryRegion xscom;
@@ -136,6 +137,7 @@ struct PnvChipClass {
     uint64_t     chip_cfam_id;
     uint64_t     cores_mask;
     uint32_t     num_phbs;
+    uint32_t     num_pecs;
 
     DeviceRealize parent_realize;
 
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index cbc3f8ed62f7..cafe7aec9aab 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -667,7 +667,7 @@ static void pnv_chip_power9_pic_print_info(PnvChip *chip, Monitor *mon)
     pnv_xive_pic_print_info(&chip9->xive, mon);
     pnv_psi_pic_print_info(&chip9->psi, mon);
 
-    for (i = 0; i < PNV9_CHIP_MAX_PEC; i++) {
+    for (i = 0; i < chip->num_pecs; i++) {
         PnvPhb4PecState *pec = &chip9->pecs[i];
         for (j = 0; j < pec->num_stacks; j++) {
             pnv_phb4_pic_print_info(&pec->stacks[j].phb, mon);
@@ -1344,15 +1344,13 @@ static void pnv_chip_power9_instance_init(Object *obj)
 
     object_initialize_child(obj, "homer", &chip9->homer, TYPE_PNV9_HOMER);
 
-    for (i = 0; i < PNV9_CHIP_MAX_PEC; i++) {
+    /* Number of PECs is the chip default */
+    chip->num_pecs = pcc->num_pecs;
+
+    for (i = 0; i < chip->num_pecs; i++) {
         object_initialize_child(obj, "pec[*]", &chip9->pecs[i],
                                 TYPE_PNV_PHB4_PEC);
     }
-
-    /*
-     * Number of PHBs is the chip default
-     */
-    chip->num_phbs = pcc->num_phbs;
 }
 
 static void pnv_chip_quad_realize(Pnv9Chip *chip9, Error **errp)
@@ -1388,7 +1386,7 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
     int i, j;
     int phb_id = 0;
 
-    for (i = 0; i < PNV9_CHIP_MAX_PEC; i++) {
+    for (i = 0; i < chip->num_pecs; i++) {
         PnvPhb4PecState *pec = &chip9->pecs[i];
         PnvPhb4PecClass *pecc = PNV_PHB4_PEC_GET_CLASS(pec);
         uint32_t pec_nest_base;
@@ -1416,8 +1414,7 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
         pnv_xscom_add_subregion(chip, pec_nest_base, &pec->nest_regs_mr);
         pnv_xscom_add_subregion(chip, pec_pci_base, &pec->pci_regs_mr);
 
-        for (j = 0; j < pec->num_stacks && phb_id < chip->num_phbs;
-             j++, phb_id++) {
+        for (j = 0; j < pec->num_stacks; j++, phb_id++) {
             PnvPhb4PecStack *stack = &pec->stacks[j];
             Object *obj = OBJECT(&stack->phb);
 
@@ -1573,7 +1570,7 @@ static void pnv_chip_power9_class_init(ObjectClass *klass, void *data)
     k->xscom_core_base = pnv_chip_power9_xscom_core_base;
     k->xscom_pcba = pnv_chip_power9_xscom_pcba;
     dc->desc = "PowerNV Chip POWER9";
-    k->num_phbs = 6;
+    k->num_pecs = PNV9_CHIP_MAX_PEC;
 
     device_class_set_parent_realize(dc, pnv_chip_power9_realize,
                                     &k->parent_realize);
-- 
2.31.1

[PATCH v2 08/19] ppc/pnv: Introduce version and device_id class atributes for PHB4 devices

[Cédric Le Goater]

It prepares ground for PHB5 which has different values.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/pci-host/pnv_phb4.h | 2 ++
 hw/pci-host/pnv_phb4_pec.c     | 2 ++
 hw/ppc/pnv.c                   | 4 ++--
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/include/hw/pci-host/pnv_phb4.h b/include/hw/pci-host/pnv_phb4.h
index 27556ae53425..b2864233641e 100644
--- a/include/hw/pci-host/pnv_phb4.h
+++ b/include/hw/pci-host/pnv_phb4.h
@@ -219,6 +219,8 @@ struct PnvPhb4PecClass {
     int compat_size;
     const char *stk_compat;
     int stk_compat_size;
+    uint64_t version;
+    uint64_t device_id;
 };
 
 #endif /* PCI_HOST_PNV_PHB4_H */
diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index 741ddc90ed8d..9f722729ac50 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -499,6 +499,8 @@ static void pnv_pec_class_init(ObjectClass *klass, void *data)
     pecc->compat_size = sizeof(compat);
     pecc->stk_compat = stk_compat;
     pecc->stk_compat_size = sizeof(stk_compat);
+    pecc->version = PNV_PHB4_VERSION;
+    pecc->device_id = PNV_PHB4_DEVICE_ID;
 }
 
 static const TypeInfo pnv_pec_type_info = {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index cafe7aec9aab..ecfe373e45e7 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1421,9 +1421,9 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
             object_property_set_int(obj, "index", phb_id, &error_fatal);
             object_property_set_int(obj, "chip-id", chip->chip_id,
                                     &error_fatal);
-            object_property_set_int(obj, "version", PNV_PHB4_VERSION,
+            object_property_set_int(obj, "version", pecc->version,
                                     &error_fatal);
-            object_property_set_int(obj, "device-id", PNV_PHB4_DEVICE_ID,
+            object_property_set_int(obj, "device-id", pecc->device_id,
                                     &error_fatal);
             object_property_set_link(obj, "stack", OBJECT(stack),
                                      &error_abort);
-- 
2.31.1

[PATCH v2 09/19] ppc/pnv: Introduce a "chip" property under the PHB4 model

[Cédric Le Goater]

And check the PEC index using the chip class.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/pci-host/pnv_phb4.h | 2 ++
 hw/pci-host/pnv_phb4_pec.c     | 7 +++++++
 hw/ppc/pnv.c                   | 2 ++
 3 files changed, 11 insertions(+)

diff --git a/include/hw/pci-host/pnv_phb4.h b/include/hw/pci-host/pnv_phb4.h
index b2864233641e..8a585c9a42f7 100644
--- a/include/hw/pci-host/pnv_phb4.h
+++ b/include/hw/pci-host/pnv_phb4.h
@@ -205,6 +205,8 @@ struct PnvPhb4PecState {
     #define PHB4_PEC_MAX_STACKS     3
     uint32_t num_stacks;
     PnvPhb4PecStack stacks[PHB4_PEC_MAX_STACKS];
+
+    PnvChip *chip;
 };
 
 
diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index 9f722729ac50..4b32b5ae6ed4 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -382,6 +382,11 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
 
     assert(pec->system_memory);
 
+    if (pec->index >= PNV_CHIP_GET_CLASS(pec->chip)->num_pecs) {
+        error_setg(errp, "invalid PEC index: %d", pec->index);
+        return;
+    }
+
     /* Create stacks */
     for (i = 0; i < pec->num_stacks; i++) {
         PnvPhb4PecStack *stack = &pec->stacks[i];
@@ -462,6 +467,8 @@ static Property pnv_pec_properties[] = {
         DEFINE_PROP_UINT32("index", PnvPhb4PecState, index, 0),
         DEFINE_PROP_UINT32("num-stacks", PnvPhb4PecState, num_stacks, 0),
         DEFINE_PROP_UINT32("chip-id", PnvPhb4PecState, chip_id, 0),
+        DEFINE_PROP_LINK("chip", PnvPhb4PecState, chip, TYPE_PNV_CHIP,
+                         PnvChip *),
         DEFINE_PROP_LINK("system-memory", PnvPhb4PecState, system_memory,
                      TYPE_MEMORY_REGION, MemoryRegion *),
         DEFINE_PROP_END_OF_LIST(),
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index ecfe373e45e7..2f8d0c19aab7 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1402,6 +1402,8 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
                                 &error_fatal);
         object_property_set_int(OBJECT(pec), "chip-id", chip->chip_id,
                                 &error_fatal);
+        object_property_set_link(OBJECT(pec), "chip", OBJECT(chip),
+                                 &error_fatal);
         object_property_set_link(OBJECT(pec), "system-memory",
                                  OBJECT(get_system_memory()), &error_abort);
         if (!qdev_realize(DEVICE(pec), NULL, errp)) {
-- 
2.31.1

[PATCH v2 10/19] ppc/pnv: Introduce a num_stack class attribute

[Cédric Le Goater]

Each PEC device of the POWER9 chip has a predefined number of stacks,
equivalent of a root port complex:

  PEC0 -> 1 stack
  PEC1 -> 2 stacks
  PEC2 -> 3 stacks

Introduce a class attribute to hold these values and remove the
"num-stacks" property.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/pci-host/pnv_phb4.h |  1 +
 hw/pci-host/pnv_phb4_pec.c     | 12 +++++++++++-
 hw/ppc/pnv.c                   |  7 -------
 3 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/include/hw/pci-host/pnv_phb4.h b/include/hw/pci-host/pnv_phb4.h
index 8a585c9a42f7..60de3031a622 100644
--- a/include/hw/pci-host/pnv_phb4.h
+++ b/include/hw/pci-host/pnv_phb4.h
@@ -223,6 +223,7 @@ struct PnvPhb4PecClass {
     int stk_compat_size;
     uint64_t version;
     uint64_t device_id;
+    const uint32_t *num_stacks;
 };
 
 #endif /* PCI_HOST_PNV_PHB4_H */
diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index 4b32b5ae6ed4..293909b5cb90 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -377,6 +377,7 @@ static void pnv_pec_instance_init(Object *obj)
 static void pnv_pec_realize(DeviceState *dev, Error **errp)
 {
     PnvPhb4PecState *pec = PNV_PHB4_PEC(dev);
+    PnvPhb4PecClass *pecc = PNV_PHB4_PEC_GET_CLASS(pec);
     char name[64];
     int i;
 
@@ -387,6 +388,8 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
         return;
     }
 
+    pec->num_stacks = pecc->num_stacks[pec->index];
+
     /* Create stacks */
     for (i = 0; i < pec->num_stacks; i++) {
         PnvPhb4PecStack *stack = &pec->stacks[i];
@@ -465,7 +468,6 @@ static int pnv_pec_dt_xscom(PnvXScomInterface *dev, void *fdt,
 
 static Property pnv_pec_properties[] = {
         DEFINE_PROP_UINT32("index", PnvPhb4PecState, index, 0),
-        DEFINE_PROP_UINT32("num-stacks", PnvPhb4PecState, num_stacks, 0),
         DEFINE_PROP_UINT32("chip-id", PnvPhb4PecState, chip_id, 0),
         DEFINE_PROP_LINK("chip", PnvPhb4PecState, chip, TYPE_PNV_CHIP,
                          PnvChip *),
@@ -484,6 +486,13 @@ static uint32_t pnv_pec_xscom_nest_base(PnvPhb4PecState *pec)
     return PNV9_XSCOM_PEC_NEST_BASE + 0x400 * pec->index;
 }
 
+/*
+ * PEC0 -> 1 stack
+ * PEC1 -> 2 stacks
+ * PEC2 -> 3 stacks
+ */
+static const uint32_t pnv_pec_num_stacks[] = { 1, 2, 3 };
+
 static void pnv_pec_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -508,6 +517,7 @@ static void pnv_pec_class_init(ObjectClass *klass, void *data)
     pecc->stk_compat_size = sizeof(stk_compat);
     pecc->version = PNV_PHB4_VERSION;
     pecc->device_id = PNV_PHB4_DEVICE_ID;
+    pecc->num_stacks = pnv_pec_num_stacks;
 }
 
 static const TypeInfo pnv_pec_type_info = {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 2f8d0c19aab7..87edbbe91e0f 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1393,13 +1393,6 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
         uint32_t pec_pci_base;
 
         object_property_set_int(OBJECT(pec), "index", i, &error_fatal);
-        /*
-         * PEC0 -> 1 stack
-         * PEC1 -> 2 stacks
-         * PEC2 -> 3 stacks
-         */
-        object_property_set_int(OBJECT(pec), "num-stacks", i + 1,
-                                &error_fatal);
         object_property_set_int(OBJECT(pec), "chip-id", chip->chip_id,
                                 &error_fatal);
         object_property_set_link(OBJECT(pec), "chip", OBJECT(chip),
-- 
2.31.1

[PATCH v2 11/19] ppc/pnv: Compute the PHB index from the PHB4 PEC model

[Cédric Le Goater]

Use the num_stacks class attribute to compute the PHB index depending
on the PEC index :

  * PEC0 provides 1 PHB  (PHB0)
  * PEC1 provides 2 PHBs (PHB1 and PHB2)
  * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)

The routine pnv_pec_phb_offset() is a bit complex but it also prepares
ground for PHB5 which has a different layout of stacks: 3 per PECs.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb4_pec.c | 16 ++++++++++++++++
 hw/ppc/pnv.c               |  4 +---
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index 293909b5cb90..a7dd4173d598 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -374,6 +374,19 @@ static void pnv_pec_instance_init(Object *obj)
     }
 }
 
+static int pnv_pec_phb_offset(PnvPhb4PecState *pec)
+{
+    PnvPhb4PecClass *pecc = PNV_PHB4_PEC_GET_CLASS(pec);
+    int index = pec->index;
+    int offset = 0;
+
+    while (index--) {
+        offset += pecc->num_stacks[index];
+    }
+
+    return offset;
+}
+
 static void pnv_pec_realize(DeviceState *dev, Error **errp)
 {
     PnvPhb4PecState *pec = PNV_PHB4_PEC(dev);
@@ -394,8 +407,10 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
     for (i = 0; i < pec->num_stacks; i++) {
         PnvPhb4PecStack *stack = &pec->stacks[i];
         Object *stk_obj = OBJECT(stack);
+        int phb_id = pnv_pec_phb_offset(pec) + i;
 
         object_property_set_int(stk_obj, "stack-no", i, &error_abort);
+        object_property_set_int(stk_obj, "phb-id", phb_id, &error_abort);
         object_property_set_link(stk_obj, "pec", OBJECT(pec), &error_abort);
         if (!qdev_realize(DEVICE(stk_obj), NULL, errp)) {
             return;
@@ -538,6 +553,7 @@ static void pnv_pec_stk_instance_init(Object *obj)
     PnvPhb4PecStack *stack = PNV_PHB4_PEC_STACK(obj);
 
     object_initialize_child(obj, "phb", &stack->phb, TYPE_PNV_PHB4);
+    object_property_add_alias(obj, "phb-id", OBJECT(&stack->phb), "index");
 }
 
 static void pnv_pec_stk_realize(DeviceState *dev, Error **errp)
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 87edbbe91e0f..284fbd50dae3 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1384,7 +1384,6 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
 {
     Pnv9Chip *chip9 = PNV9_CHIP(chip);
     int i, j;
-    int phb_id = 0;
 
     for (i = 0; i < chip->num_pecs; i++) {
         PnvPhb4PecState *pec = &chip9->pecs[i];
@@ -1409,11 +1408,10 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
         pnv_xscom_add_subregion(chip, pec_nest_base, &pec->nest_regs_mr);
         pnv_xscom_add_subregion(chip, pec_pci_base, &pec->pci_regs_mr);
 
-        for (j = 0; j < pec->num_stacks; j++, phb_id++) {
+        for (j = 0; j < pec->num_stacks; j++) {
             PnvPhb4PecStack *stack = &pec->stacks[j];
             Object *obj = OBJECT(&stack->phb);
 
-            object_property_set_int(obj, "index", phb_id, &error_fatal);
             object_property_set_int(obj, "chip-id", chip->chip_id,
                                     &error_fatal);
             object_property_set_int(obj, "version", pecc->version,
-- 
2.31.1

[PATCH v2 12/19] ppc/pnv: Remove "system-memory" property from PHB4 PEC

[Cédric Le Goater]

This is not useful and will be in the way for support of user created
PHB4 devices.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb4_pec.c | 6 +-----
 hw/ppc/pnv.c               | 2 --
 2 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index a7dd4173d598..dfed2af0f7df 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -124,7 +124,7 @@ static uint64_t pnv_pec_stk_nest_xscom_read(void *opaque, hwaddr addr,
 static void pnv_pec_stk_update_map(PnvPhb4PecStack *stack)
 {
     PnvPhb4PecState *pec = stack->pec;
-    MemoryRegion *sysmem = pec->system_memory;
+    MemoryRegion *sysmem = get_system_memory();
     uint64_t bar_en = stack->nest_regs[PEC_NEST_STK_BAR_EN];
     uint64_t bar, mask, size;
     char name[64];
@@ -394,8 +394,6 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
     char name[64];
     int i;
 
-    assert(pec->system_memory);
-
     if (pec->index >= PNV_CHIP_GET_CLASS(pec->chip)->num_pecs) {
         error_setg(errp, "invalid PEC index: %d", pec->index);
         return;
@@ -486,8 +484,6 @@ static Property pnv_pec_properties[] = {
         DEFINE_PROP_UINT32("chip-id", PnvPhb4PecState, chip_id, 0),
         DEFINE_PROP_LINK("chip", PnvPhb4PecState, chip, TYPE_PNV_CHIP,
                          PnvChip *),
-        DEFINE_PROP_LINK("system-memory", PnvPhb4PecState, system_memory,
-                     TYPE_MEMORY_REGION, MemoryRegion *),
         DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 284fbd50dae3..c59a4a078cd0 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1396,8 +1396,6 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
                                 &error_fatal);
         object_property_set_link(OBJECT(pec), "chip", OBJECT(chip),
                                  &error_fatal);
-        object_property_set_link(OBJECT(pec), "system-memory",
-                                 OBJECT(get_system_memory()), &error_abort);
         if (!qdev_realize(DEVICE(pec), NULL, errp)) {
             return;
         }
-- 
2.31.1

[PATCH v2 13/19] ppc/pnv: Move realize of PEC stacks under the PEC model

[Cédric Le Goater]

This change will help us providing support for user created PHB4
devices.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb4_pec.c | 34 ++++++++++++++++++++++++++++++----
 hw/ppc/pnv.c               | 37 ++++---------------------------------
 2 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index dfed2af0f7df..24a3adcae326 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -556,6 +556,10 @@ static void pnv_pec_stk_realize(DeviceState *dev, Error **errp)
 {
     PnvPhb4PecStack *stack = PNV_PHB4_PEC_STACK(dev);
     PnvPhb4PecState *pec = stack->pec;
+    PnvPhb4PecClass *pecc = PNV_PHB4_PEC_GET_CLASS(pec);
+    PnvChip *chip = pec->chip;
+    uint32_t pec_nest_base;
+    uint32_t pec_pci_base;
     char name[64];
 
     assert(pec);
@@ -579,10 +583,32 @@ static void pnv_pec_stk_realize(DeviceState *dev, Error **errp)
     pnv_xscom_region_init(&stack->phb_regs_mr, OBJECT(&stack->phb),
                           &pnv_phb4_xscom_ops, &stack->phb, name, 0x40);
 
-    /*
-     * Let the machine/chip realize the PHB object to customize more
-     * easily some fields
-     */
+    object_property_set_int(OBJECT(&stack->phb), "chip-id", pec->chip_id,
+                            &error_fatal);
+    object_property_set_int(OBJECT(&stack->phb), "version", pecc->version,
+                            &error_fatal);
+    object_property_set_int(OBJECT(&stack->phb), "device-id", pecc->device_id,
+                            &error_fatal);
+    object_property_set_link(OBJECT(&stack->phb), "stack", OBJECT(stack),
+                             &error_abort);
+    if (!sysbus_realize(SYS_BUS_DEVICE(&stack->phb), errp)) {
+        return;
+    }
+
+    pec_nest_base = pecc->xscom_nest_base(pec);
+    pec_pci_base = pecc->xscom_pci_base(pec);
+
+    /* Populate the XSCOM address space. */
+    pnv_xscom_add_subregion(chip,
+                            pec_nest_base + 0x40 * (stack->stack_no + 1),
+                            &stack->nest_regs_mr);
+    pnv_xscom_add_subregion(chip,
+                            pec_pci_base + 0x40 * (stack->stack_no + 1),
+                            &stack->pci_regs_mr);
+    pnv_xscom_add_subregion(chip,
+                            pec_pci_base + PNV9_XSCOM_PEC_PCI_STK0 +
+                            0x40 * stack->stack_no,
+                            &stack->phb_regs_mr);
 }
 
 static Property pnv_pec_stk_properties[] = {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index c59a4a078cd0..08b037f4e753 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1380,10 +1380,10 @@ static void pnv_chip_quad_realize(Pnv9Chip *chip9, Error **errp)
     }
 }
 
-static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
+static void pnv_chip_power9_pec_realize(PnvChip *chip, Error **errp)
 {
     Pnv9Chip *chip9 = PNV9_CHIP(chip);
-    int i, j;
+    int i;
 
     for (i = 0; i < chip->num_pecs; i++) {
         PnvPhb4PecState *pec = &chip9->pecs[i];
@@ -1405,35 +1405,6 @@ static void pnv_chip_power9_phb_realize(PnvChip *chip, Error **errp)
 
         pnv_xscom_add_subregion(chip, pec_nest_base, &pec->nest_regs_mr);
         pnv_xscom_add_subregion(chip, pec_pci_base, &pec->pci_regs_mr);
-
-        for (j = 0; j < pec->num_stacks; j++) {
-            PnvPhb4PecStack *stack = &pec->stacks[j];
-            Object *obj = OBJECT(&stack->phb);
-
-            object_property_set_int(obj, "chip-id", chip->chip_id,
-                                    &error_fatal);
-            object_property_set_int(obj, "version", pecc->version,
-                                    &error_fatal);
-            object_property_set_int(obj, "device-id", pecc->device_id,
-                                    &error_fatal);
-            object_property_set_link(obj, "stack", OBJECT(stack),
-                                     &error_abort);
-            if (!sysbus_realize(SYS_BUS_DEVICE(obj), errp)) {
-                return;
-            }
-
-            /* Populate the XSCOM address space. */
-            pnv_xscom_add_subregion(chip,
-                                   pec_nest_base + 0x40 * (stack->stack_no + 1),
-                                   &stack->nest_regs_mr);
-            pnv_xscom_add_subregion(chip,
-                                    pec_pci_base + 0x40 * (stack->stack_no + 1),
-                                    &stack->pci_regs_mr);
-            pnv_xscom_add_subregion(chip,
-                                    pec_pci_base + PNV9_XSCOM_PEC_PCI_STK0 +
-                                    0x40 * stack->stack_no,
-                                    &stack->phb_regs_mr);
-        }
     }
 }
 
@@ -1529,8 +1500,8 @@ static void pnv_chip_power9_realize(DeviceState *dev, Error **errp)
     memory_region_add_subregion(get_system_memory(), PNV9_HOMER_BASE(chip),
                                 &chip9->homer.regs);
 
-    /* PHBs */
-    pnv_chip_power9_phb_realize(chip, &local_err);
+    /* PEC PHBs */
+    pnv_chip_power9_pec_realize(chip, &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
         return;
-- 
2.31.1

[PATCH v2 14/19] ppc/pnv: Use QOM hierarchy to scan PEC PHB4 devices

[Cédric Le Goater]

When -nodefaults is supported for PHB4 devices, the pecs array under
the chip will be empty. This will break the 'info pic' HMP command.

Do a QOM loop on the chip children and look for PEC PHB4 devices
instead.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/ppc/pnv.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 08b037f4e753..9de8b8353014 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -659,20 +659,26 @@ static void pnv_chip_power8_pic_print_info(PnvChip *chip, Monitor *mon)
                          pnv_chip_power8_pic_print_info_child, mon);
 }
 
+static int pnv_chip_power9_pic_print_info_child(Object *child, void *opaque)
+{
+    Monitor *mon = opaque;
+    PnvPHB4 *phb4 = (PnvPHB4 *) object_dynamic_cast(child, TYPE_PNV_PHB4);
+
+    if (phb4) {
+        pnv_phb4_pic_print_info(phb4, mon);
+    }
+    return 0;
+}
+
 static void pnv_chip_power9_pic_print_info(PnvChip *chip, Monitor *mon)
 {
     Pnv9Chip *chip9 = PNV9_CHIP(chip);
-    int i, j;
 
     pnv_xive_pic_print_info(&chip9->xive, mon);
     pnv_psi_pic_print_info(&chip9->psi, mon);
 
-    for (i = 0; i < chip->num_pecs; i++) {
-        PnvPhb4PecState *pec = &chip9->pecs[i];
-        for (j = 0; j < pec->num_stacks; j++) {
-            pnv_phb4_pic_print_info(&pec->stacks[j].phb, mon);
-        }
-    }
+    object_child_foreach_recursive(OBJECT(chip),
+                         pnv_chip_power9_pic_print_info_child, mon);
 }
 
 static uint64_t pnv_chip_power8_xscom_core_base(PnvChip *chip,
-- 
2.31.1

Re: [PATCH v2 14/19] ppc/pnv: Use QOM hierarchy to scan PEC PHB4 devices

[Daniel Henrique Barboza]

On 12/13/21 10:28, Cédric Le Goater wrote:
> When -nodefaults is supported for PHB4 devices, the pecs array under
> the chip will be empty. This will break the 'info pic' HMP command.
> 
> Do a QOM loop on the chip children and look for PEC PHB4 devices
> instead.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>

>   hw/ppc/pnv.c | 20 +++++++++++++-------
>   1 file changed, 13 insertions(+), 7 deletions(-)
> 
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 08b037f4e753..9de8b8353014 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -659,20 +659,26 @@ static void pnv_chip_power8_pic_print_info(PnvChip *chip, Monitor *mon)
>                            pnv_chip_power8_pic_print_info_child, mon);
>   }
>   
> +static int pnv_chip_power9_pic_print_info_child(Object *child, void *opaque)
> +{
> +    Monitor *mon = opaque;
> +    PnvPHB4 *phb4 = (PnvPHB4 *) object_dynamic_cast(child, TYPE_PNV_PHB4);
> +
> +    if (phb4) {
> +        pnv_phb4_pic_print_info(phb4, mon);
> +    }
> +    return 0;
> +}
> +
>   static void pnv_chip_power9_pic_print_info(PnvChip *chip, Monitor *mon)
>   {
>       Pnv9Chip *chip9 = PNV9_CHIP(chip);
> -    int i, j;
>   
>       pnv_xive_pic_print_info(&chip9->xive, mon);
>       pnv_psi_pic_print_info(&chip9->psi, mon);
>   
> -    for (i = 0; i < chip->num_pecs; i++) {
> -        PnvPhb4PecState *pec = &chip9->pecs[i];
> -        for (j = 0; j < pec->num_stacks; j++) {
> -            pnv_phb4_pic_print_info(&pec->stacks[j].phb, mon);
> -        }
> -    }
> +    object_child_foreach_recursive(OBJECT(chip),
> +                         pnv_chip_power9_pic_print_info_child, mon);
>   }
>   
>   static uint64_t pnv_chip_power8_xscom_core_base(PnvChip *chip,
> 

[PATCH v2 15/19] ppc/pnv: Introduce support for user created PHB3 devices

[Cédric Le Goater]

PHB3 devices and PCI devices can now be added to the powernv8 machine
using :

  -device pnv-phb3,chip-id=0,index=1 \
  -device nec-usb-xhci,bus=pci.1,addr=0x0

The 'index' property identifies the PHB3 in the chip. In case of user
created devices, a lookup on 'chip-id' is required to assign the
owning chip.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/pnv.h   |  2 ++
 hw/pci-host/pnv_phb3.c | 11 ++++++++++-
 hw/ppc/pnv.c           | 23 ++++++++++++++++++-----
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
index c781525277db..9b06a9e9948d 100644
--- a/include/hw/ppc/pnv.h
+++ b/include/hw/ppc/pnv.h
@@ -217,6 +217,8 @@ struct PnvMachineState {
     hwaddr       fw_load_addr;
 };
 
+PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id);
+
 #define PNV_FDT_ADDR          0x01000000
 #define PNV_TIMEBASE_FREQ     512000000ULL
 
diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index 9c4451ca0d1c..4a022de7c1dc 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -993,6 +993,15 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
     PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
     int i;
 
+    /* User created devices */
+    if (!phb->chip) {
+        phb->chip = pnv_get_chip(pnv, phb->chip_id);
+        if (!phb->chip) {
+            error_setg(errp, "invalid chip id: %d", phb->chip_id);
+            return;
+        }
+    }
+
     if (phb->phb_id >= PNV_CHIP_GET_CLASS(phb->chip)->num_phbs) {
         error_setg(errp, "invalid PHB index: %d", phb->phb_id);
         return;
@@ -1105,7 +1114,7 @@ static void pnv_phb3_class_init(ObjectClass *klass, void *data)
     dc->realize = pnv_phb3_realize;
     device_class_set_props(dc, pnv_phb3_properties);
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
-    dc->user_creatable = false;
+    dc->user_creatable = true;
 }
 
 static const TypeInfo pnv_phb3_type_info = {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 9de8b8353014..d134eca0ee9d 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1117,14 +1117,14 @@ static void pnv_chip_power8_instance_init(Object *obj)
 
     object_initialize_child(obj, "homer", &chip8->homer, TYPE_PNV8_HOMER);
 
-    for (i = 0; i < pcc->num_phbs; i++) {
+    if (defaults_enabled()) {
+        chip->num_phbs = pcc->num_phbs;
+    }
+
+    for (i = 0; i < chip->num_phbs; i++) {
         object_initialize_child(obj, "phb[*]", &chip8->phbs[i], TYPE_PNV_PHB3);
     }
 
-    /*
-     * Number of PHBs is the chip default
-     */
-    chip->num_phbs = pcc->num_phbs;
 }
 
 static void pnv_chip_icp_realize(Pnv8Chip *chip8, Error **errp)
@@ -1806,6 +1806,19 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
     return NULL;
 }
 
+PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id)
+{
+    int i;
+
+    for (i = 0; i < pnv->num_chips; i++) {
+        PnvChip *chip = pnv->chips[i];
+        if (chip->chip_id == chip_id) {
+            return chip;
+        }
+    }
+    return NULL;
+}
+
 static int pnv_ics_resend_child(Object *child, void *opaque)
 {
     PnvPHB3 *phb3 = (PnvPHB3 *) object_dynamic_cast(child, TYPE_PNV_PHB3);
-- 
2.31.1

[PATCH v2 16/19] ppc/pnv: Reparent user created PHB3 devices to the PnvChip

[Cédric Le Goater]

The powernv machine uses the object hierarchy to populate the device
tree and each device should be parented to the chip it belongs to.
This is not the case for user created devices which are parented to
the container "/unattached".

Make sure a PHB3 device is parented to its chip by reparenting the
object if necessary.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/pnv.h   |  1 +
 hw/pci-host/pnv_phb3.c |  6 ++++++
 hw/ppc/pnv.c           | 17 +++++++++++++++++
 3 files changed, 24 insertions(+)

diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
index 9b06a9e9948d..ca27bd39f0ac 100644
--- a/include/hw/ppc/pnv.h
+++ b/include/hw/ppc/pnv.h
@@ -177,6 +177,7 @@ DECLARE_INSTANCE_CHECKER(PnvChip, PNV_CHIP_POWER10,
                          TYPE_PNV_CHIP_POWER10)
 
 PowerPCCPU *pnv_chip_find_cpu(PnvChip *chip, uint32_t pir);
+void pnv_chip_parent_fixup(PnvChip *chip, Object *obj, int index);
 
 #define TYPE_PNV_MACHINE       MACHINE_TYPE_NAME("powernv")
 typedef struct PnvMachineClass PnvMachineClass;
diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index 4a022de7c1dc..62a184c9ddc9 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -1000,6 +1000,12 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
             error_setg(errp, "invalid chip id: %d", phb->chip_id);
             return;
         }
+
+        /*
+         * Reparent user created devices to the chip to build
+         * correctly the device tree.
+         */
+        pnv_chip_parent_fixup(phb->chip, OBJECT(phb), phb->phb_id);
     }
 
     if (phb->phb_id >= PNV_CHIP_GET_CLASS(phb->chip)->num_phbs) {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index d134eca0ee9d..ac048a0a5d24 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1806,6 +1806,23 @@ static ICSState *pnv_ics_get(XICSFabric *xi, int irq)
     return NULL;
 }
 
+void pnv_chip_parent_fixup(PnvChip *chip, Object *obj, int index)
+{
+    Object *parent = OBJECT(chip);
+    g_autofree char *default_id =
+        g_strdup_printf("%s[%d]", object_get_typename(obj), index);
+
+    if (obj->parent == parent) {
+        return;
+    }
+
+    object_ref(obj);
+    object_unparent(obj);
+    object_property_add_child(
+        parent, DEVICE(obj)->id ? DEVICE(obj)->id : default_id, obj);
+    object_unref(obj);
+}
+
 PnvChip *pnv_get_chip(PnvMachineState *pnv, uint32_t chip_id)
 {
     int i;
-- 
2.31.1

[PATCH v2 17/19] ppc/pnv: Complete user created PHB3 devices

[Cédric Le Goater]

PHB3s ared SysBus devices and should be allowed to be dynamically
created.

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb3.c | 9 +++++++++
 hw/ppc/pnv.c           | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index 62a184c9ddc9..e51ae4c969e8 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -995,6 +995,9 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
 
     /* User created devices */
     if (!phb->chip) {
+        Error *local_err = NULL;
+        BusState *s;
+
         phb->chip = pnv_get_chip(pnv, phb->chip_id);
         if (!phb->chip) {
             error_setg(errp, "invalid chip id: %d", phb->chip_id);
@@ -1006,6 +1009,12 @@ static void pnv_phb3_realize(DeviceState *dev, Error **errp)
          * correctly the device tree.
          */
         pnv_chip_parent_fixup(phb->chip, OBJECT(phb), phb->phb_id);
+
+        s = qdev_get_parent_bus(DEVICE(phb->chip));
+        if (!qdev_set_parent_bus(DEVICE(phb), s, &local_err)) {
+            error_propagate(errp, local_err);
+            return;
+        }
     }
 
     if (phb->phb_id >= PNV_CHIP_GET_CLASS(phb->chip)->num_phbs) {
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index ac048a0a5d24..9c45aae1befe 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1933,6 +1933,8 @@ static void pnv_machine_power8_class_init(ObjectClass *oc, void *data)
 
     pmc->compat = compat;
     pmc->compat_size = sizeof(compat);
+
+    machine_class_allow_dynamic_sysbus_dev(mc, TYPE_PNV_PHB3);
 }
 
 static void pnv_machine_power9_class_init(ObjectClass *oc, void *data)
-- 
2.31.1

[PATCH v2 18/19] ppc/pnv: Introduce support for user created PHB4 devices

[Cédric Le Goater]

PHB4 devices and PCI devices can now be added to the powernv9 machine
using:

  -device pnv-phb4-pec,chip-id=0,index=0
  -device nec-usb-xhci,bus=pci.0,addr=0x0

In case of user created devices, a lookup on 'chip-id' is required to
assign the owning chip.

To be noted, that the PEC PHB4 devices can add more than one PHB4
devices:

  * PEC0 provides 1 PHB  (PHB0)
  * PEC1 provides 2 PHBs (PHB1 and PHB2)
  * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)

Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 hw/pci-host/pnv_phb4_pec.c | 19 ++++++++++++++++++-
 hw/ppc/pnv.c               |  5 +++--
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/hw/pci-host/pnv_phb4_pec.c b/hw/pci-host/pnv_phb4_pec.c
index 24a3adcae326..4fbb8061455d 100644
--- a/hw/pci-host/pnv_phb4_pec.c
+++ b/hw/pci-host/pnv_phb4_pec.c
@@ -394,6 +394,17 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
     char name[64];
     int i;
 
+    /* User created devices */
+    if (!pec->chip) {
+        PnvMachineState *pnv = PNV_MACHINE(qdev_get_machine());
+
+        pec->chip = pnv_get_chip(pnv, pec->chip_id);
+        if (!pec->chip) {
+            error_setg(errp, "invalid chip id: %d", pec->chip_id);
+            return;
+        }
+    }
+
     if (pec->index >= PNV_CHIP_GET_CLASS(pec->chip)->num_pecs) {
         error_setg(errp, "invalid PEC index: %d", pec->index);
         return;
@@ -401,6 +412,12 @@ static void pnv_pec_realize(DeviceState *dev, Error **errp)
 
     pec->num_stacks = pecc->num_stacks[pec->index];
 
+    /*
+     * Reparent user created devices to the chip to build correctly
+     * the device tree.
+     */
+    pnv_chip_parent_fixup(pec->chip, OBJECT(pec), pec->index);
+
     /* Create stacks */
     for (i = 0; i < pec->num_stacks; i++) {
         PnvPhb4PecStack *stack = &pec->stacks[i];
@@ -516,7 +533,7 @@ static void pnv_pec_class_init(ObjectClass *klass, void *data)
 
     dc->realize = pnv_pec_realize;
     device_class_set_props(dc, pnv_pec_properties);
-    dc->user_creatable = false;
+    dc->user_creatable = true;
 
     pecc->xscom_nest_base = pnv_pec_xscom_nest_base;
     pecc->xscom_pci_base  = pnv_pec_xscom_pci_base;
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 9c45aae1befe..2b027e299d27 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1350,8 +1350,9 @@ static void pnv_chip_power9_instance_init(Object *obj)
 
     object_initialize_child(obj, "homer", &chip9->homer, TYPE_PNV9_HOMER);
 
-    /* Number of PECs is the chip default */
-    chip->num_pecs = pcc->num_pecs;
+    if (defaults_enabled()) {
+        chip->num_pecs = pcc->num_pecs;
+    }
 
     for (i = 0; i < chip->num_pecs; i++) {
         object_initialize_child(obj, "pec[*]", &chip9->pecs[i],
-- 
2.31.1

[PATCH v2 19/19] ppc/pnv: Move num_phbs under Pnv8Chip

[Cédric Le Goater]

It is not used elsewhere so that's where it belongs.

Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
 include/hw/ppc/pnv.h | 4 ++--
 hw/ppc/pnv.c         | 7 +++----
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
index ca27bd39f0ac..251c9854329d 100644
--- a/include/hw/ppc/pnv.h
+++ b/include/hw/ppc/pnv.h
@@ -52,7 +52,6 @@ struct PnvChip {
     uint64_t     cores_mask;
     PnvCore      **cores;
 
-    uint32_t     num_phbs;
     uint32_t     num_pecs;
 
     MemoryRegion xscom_mmio;
@@ -82,6 +81,7 @@ struct Pnv8Chip {
 
 #define PNV8_CHIP_PHB3_MAX 4
     PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
+    uint32_t     num_phbs;
 
     XICSFabric    *xics;
 };
@@ -136,8 +136,8 @@ struct PnvChipClass {
     /*< public >*/
     uint64_t     chip_cfam_id;
     uint64_t     cores_mask;
-    uint32_t     num_phbs;
     uint32_t     num_pecs;
+    uint32_t     num_phbs;
 
     DeviceRealize parent_realize;
 
diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index 2b027e299d27..8a3732c982e5 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -1099,7 +1099,6 @@ static void pnv_chip_power10_intc_print_info(PnvChip *chip, PowerPCCPU *cpu,
 
 static void pnv_chip_power8_instance_init(Object *obj)
 {
-    PnvChip *chip = PNV_CHIP(obj);
     Pnv8Chip *chip8 = PNV8_CHIP(obj);
     PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
     int i;
@@ -1118,10 +1117,10 @@ static void pnv_chip_power8_instance_init(Object *obj)
     object_initialize_child(obj, "homer", &chip8->homer, TYPE_PNV8_HOMER);
 
     if (defaults_enabled()) {
-        chip->num_phbs = pcc->num_phbs;
+        chip8->num_phbs = pcc->num_phbs;
     }
 
-    for (i = 0; i < chip->num_phbs; i++) {
+    for (i = 0; i < chip8->num_phbs; i++) {
         object_initialize_child(obj, "phb[*]", &chip8->phbs[i], TYPE_PNV_PHB3);
     }
 
@@ -1239,7 +1238,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
                                 &chip8->homer.regs);
 
     /* PHB3 controllers */
-    for (i = 0; i < chip->num_phbs; i++) {
+    for (i = 0; i < chip8->num_phbs; i++) {
         PnvPHB3 *phb = &chip8->phbs[i];
 
         object_property_set_int(OBJECT(phb), "index", i, &error_fatal);
-- 
2.31.1

Re: [PATCH v2 19/19] ppc/pnv: Move num_phbs under Pnv8Chip

[Daniel Henrique Barboza]

On 12/13/21 10:28, Cédric Le Goater wrote:
> It is not used elsewhere so that's where it belongs.
> 
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---

I am/was using this patch and didn't provide my r-b on it:


Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>


>   include/hw/ppc/pnv.h | 4 ++--
>   hw/ppc/pnv.c         | 7 +++----
>   2 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
> index ca27bd39f0ac..251c9854329d 100644
> --- a/include/hw/ppc/pnv.h
> +++ b/include/hw/ppc/pnv.h
> @@ -52,7 +52,6 @@ struct PnvChip {
>       uint64_t     cores_mask;
>       PnvCore      **cores;
>   
> -    uint32_t     num_phbs;
>       uint32_t     num_pecs;
>   
>       MemoryRegion xscom_mmio;
> @@ -82,6 +81,7 @@ struct Pnv8Chip {
>   
>   #define PNV8_CHIP_PHB3_MAX 4
>       PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
> +    uint32_t     num_phbs;
>   
>       XICSFabric    *xics;
>   };
> @@ -136,8 +136,8 @@ struct PnvChipClass {
>       /*< public >*/
>       uint64_t     chip_cfam_id;
>       uint64_t     cores_mask;
> -    uint32_t     num_phbs;
>       uint32_t     num_pecs;
> +    uint32_t     num_phbs;
>   
>       DeviceRealize parent_realize;
>   
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index 2b027e299d27..8a3732c982e5 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -1099,7 +1099,6 @@ static void pnv_chip_power10_intc_print_info(PnvChip *chip, PowerPCCPU *cpu,
>   
>   static void pnv_chip_power8_instance_init(Object *obj)
>   {
> -    PnvChip *chip = PNV_CHIP(obj);
>       Pnv8Chip *chip8 = PNV8_CHIP(obj);
>       PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
>       int i;
> @@ -1118,10 +1117,10 @@ static void pnv_chip_power8_instance_init(Object *obj)
>       object_initialize_child(obj, "homer", &chip8->homer, TYPE_PNV8_HOMER);
>   
>       if (defaults_enabled()) {
> -        chip->num_phbs = pcc->num_phbs;
> +        chip8->num_phbs = pcc->num_phbs;
>       }
>   
> -    for (i = 0; i < chip->num_phbs; i++) {
> +    for (i = 0; i < chip8->num_phbs; i++) {
>           object_initialize_child(obj, "phb[*]", &chip8->phbs[i], TYPE_PNV_PHB3);
>       }
>   
> @@ -1239,7 +1238,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>                                   &chip8->homer.regs);
>   
>       /* PHB3 controllers */
> -    for (i = 0; i < chip->num_phbs; i++) {
> +    for (i = 0; i < chip8->num_phbs; i++) {
>           PnvPHB3 *phb = &chip8->phbs[i];
>   
>           object_property_set_int(OBJECT(phb), "index", i, &error_fatal);

Re: [PATCH v2 19/19] ppc/pnv: Move num_phbs under Pnv8Chip

[Cédric Le Goater]

On 1/4/22 10:53, Daniel Henrique Barboza wrote:
> 
> 
> On 12/13/21 10:28, Cédric Le Goater wrote:
>> It is not used elsewhere so that's where it belongs.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
> 
> I am/was using this patch and didn't provide my r-b on it:
> 
> 
> Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com>

Yes. Just resend with your next series.

Thanks,

C.


> 
>>   include/hw/ppc/pnv.h | 4 ++--
>>   hw/ppc/pnv.c         | 7 +++----
>>   2 files changed, 5 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/hw/ppc/pnv.h b/include/hw/ppc/pnv.h
>> index ca27bd39f0ac..251c9854329d 100644
>> --- a/include/hw/ppc/pnv.h
>> +++ b/include/hw/ppc/pnv.h
>> @@ -52,7 +52,6 @@ struct PnvChip {
>>       uint64_t     cores_mask;
>>       PnvCore      **cores;
>> -    uint32_t     num_phbs;
>>       uint32_t     num_pecs;
>>       MemoryRegion xscom_mmio;
>> @@ -82,6 +81,7 @@ struct Pnv8Chip {
>>   #define PNV8_CHIP_PHB3_MAX 4
>>       PnvPHB3      phbs[PNV8_CHIP_PHB3_MAX];
>> +    uint32_t     num_phbs;
>>       XICSFabric    *xics;
>>   };
>> @@ -136,8 +136,8 @@ struct PnvChipClass {
>>       /*< public >*/
>>       uint64_t     chip_cfam_id;
>>       uint64_t     cores_mask;
>> -    uint32_t     num_phbs;
>>       uint32_t     num_pecs;
>> +    uint32_t     num_phbs;
>>       DeviceRealize parent_realize;
>> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
>> index 2b027e299d27..8a3732c982e5 100644
>> --- a/hw/ppc/pnv.c
>> +++ b/hw/ppc/pnv.c
>> @@ -1099,7 +1099,6 @@ static void pnv_chip_power10_intc_print_info(PnvChip *chip, PowerPCCPU *cpu,
>>   static void pnv_chip_power8_instance_init(Object *obj)
>>   {
>> -    PnvChip *chip = PNV_CHIP(obj);
>>       Pnv8Chip *chip8 = PNV8_CHIP(obj);
>>       PnvChipClass *pcc = PNV_CHIP_GET_CLASS(obj);
>>       int i;
>> @@ -1118,10 +1117,10 @@ static void pnv_chip_power8_instance_init(Object *obj)
>>       object_initialize_child(obj, "homer", &chip8->homer, TYPE_PNV8_HOMER);
>>       if (defaults_enabled()) {
>> -        chip->num_phbs = pcc->num_phbs;
>> +        chip8->num_phbs = pcc->num_phbs;
>>       }
>> -    for (i = 0; i < chip->num_phbs; i++) {
>> +    for (i = 0; i < chip8->num_phbs; i++) {
>>           object_initialize_child(obj, "phb[*]", &chip8->phbs[i], TYPE_PNV_PHB3);
>>       }
>> @@ -1239,7 +1238,7 @@ static void pnv_chip_power8_realize(DeviceState *dev, Error **errp)
>>                                   &chip8->homer.regs);
>>       /* PHB3 controllers */
>> -    for (i = 0; i < chip->num_phbs; i++) {
>> +    for (i = 0; i < chip8->num_phbs; i++) {
>>           PnvPHB3 *phb = &chip8->phbs[i];
>>           object_property_set_int(OBJECT(phb), "index", i, &error_fatal);

Re: [PATCH v2 00/19] ppc/pnv: Add support for user created PHB3/PHB4 devices

[Cédric Le Goater]

On 12/13/21 14:28, Cédric Le Goater wrote:
> Hello,
> 
> On the POWER8 processor, powernv8 machine, PHB3 devices can simply be
> created with :
> 
>     -device pnv-phb3,chip-id=0,index=1
> 
> with a maximum of 3 PHB3s per chip, each PHB3 adding a new PCIe bus.
> 
> On the POWER9 processor, powernv9 machine, the logic is different. The
> the chip comes with 3 PHB4 PECs (PCI Express Controller) and each PEC
> can have several PHBs :
> 
>    * PEC0 provides 1 PHB  (PHB0)
>    * PEC1 provides 2 PHBs (PHB1 and PHB2)
>    * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)
> 
> The PEC devices can be created with :
> 
>     -device pnv-phb4-pec,chip-id=0,index=1
> 
> And the number of added PHB4 devices depends on the PEC index. Each
> PHB4 adds a new PCIe bus.
> 
> The following changes are mostly cleanups and improvements of the
> PHB3/4 realize routines to enable support. One important change is
> related to the way the powernv machine populates the device tree. It
> depends on the object hierarchy and it is necessary to reparent user
> created devices to the chip they belong to (see PATCH 5). PHB3 is a
> little more sophisticated because of its SysBusDevice nature (see
> PATCH 6).
> 
> It would be preferable for libvirt and user to add one PHB4 (one PCIe
> bus) at a time but that's another step. The plan is to merge real soon
> the first patches which are required cleanups of the models and give
> some more time for the last ones.

Applied patches 1-14 which are simple cleanups to ppc-next.

Thanks,

C.

Re: [PATCH v2 00/19] ppc/pnv: Add support for user created PHB3/PHB4 devices

[Daniel Henrique Barboza]

On 12/15/21 13:56, Cédric Le Goater wrote:
> On 12/13/21 14:28, Cédric Le Goater wrote:
>> Hello,
>>
>> On the POWER8 processor, powernv8 machine, PHB3 devices can simply be
>> created with :
>>
>>     -device pnv-phb3,chip-id=0,index=1
>>
>> with a maximum of 3 PHB3s per chip, each PHB3 adding a new PCIe bus.
>>
>> On the POWER9 processor, powernv9 machine, the logic is different. The
>> the chip comes with 3 PHB4 PECs (PCI Express Controller) and each PEC
>> can have several PHBs :
>>
>>    * PEC0 provides 1 PHB  (PHB0)
>>    * PEC1 provides 2 PHBs (PHB1 and PHB2)
>>    * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)
>>
>> The PEC devices can be created with :
>>
>>     -device pnv-phb4-pec,chip-id=0,index=1
>>
>> And the number of added PHB4 devices depends on the PEC index. Each
>> PHB4 adds a new PCIe bus.
>>
>> The following changes are mostly cleanups and improvements of the
>> PHB3/4 realize routines to enable support. One important change is
>> related to the way the powernv machine populates the device tree. It
>> depends on the object hierarchy and it is necessary to reparent user
>> created devices to the chip they belong to (see PATCH 5). PHB3 is a
>> little more sophisticated because of its SysBusDevice nature (see
>> PATCH 6).
>>
>> It would be preferable for libvirt and user to add one PHB4 (one PCIe
>> bus) at a time but that's another step. The plan is to merge real soon
>> the first patches which are required cleanups of the models and give
>> some more time for the last ones.
> 
> Applied patches 1-14 which are simple cleanups to ppc-next.

Did you also push patches 15-19? Or these were the ones that you decided to
discard?


Thanks,


Daniel

> 
> Thanks,
> 
> C.

Re: [PATCH v2 00/19] ppc/pnv: Add support for user created PHB3/PHB4 devices

[Cédric Le Goater]

On 1/4/22 10:56, Daniel Henrique Barboza wrote:
> 
> 
> On 12/15/21 13:56, Cédric Le Goater wrote:
>> On 12/13/21 14:28, Cédric Le Goater wrote:
>>> Hello,
>>>
>>> On the POWER8 processor, powernv8 machine, PHB3 devices can simply be
>>> created with :
>>>
>>>     -device pnv-phb3,chip-id=0,index=1
>>>
>>> with a maximum of 3 PHB3s per chip, each PHB3 adding a new PCIe bus.
>>>
>>> On the POWER9 processor, powernv9 machine, the logic is different. The
>>> the chip comes with 3 PHB4 PECs (PCI Express Controller) and each PEC
>>> can have several PHBs :
>>>
>>>    * PEC0 provides 1 PHB  (PHB0)
>>>    * PEC1 provides 2 PHBs (PHB1 and PHB2)
>>>    * PEC2 provides 3 PHBs (PHB3, PHB4 and PHB5)
>>>
>>> The PEC devices can be created with :
>>>
>>>     -device pnv-phb4-pec,chip-id=0,index=1
>>>
>>> And the number of added PHB4 devices depends on the PEC index. Each
>>> PHB4 adds a new PCIe bus.
>>>
>>> The following changes are mostly cleanups and improvements of the
>>> PHB3/4 realize routines to enable support. One important change is
>>> related to the way the powernv machine populates the device tree. It
>>> depends on the object hierarchy and it is necessary to reparent user
>>> created devices to the chip they belong to (see PATCH 5). PHB3 is a
>>> little more sophisticated because of its SysBusDevice nature (see
>>> PATCH 6).
>>>
>>> It would be preferable for libvirt and user to add one PHB4 (one PCIe
>>> bus) at a time but that's another step. The plan is to merge real soon
>>> the first patches which are required cleanups of the models and give
>>> some more time for the last ones.
>>
>> Applied patches 1-14 which are simple cleanups to ppc-next.
> 
> Did you also push patches 15-19? Or these were the ones that you decided to
> discard?

1-14 are now merged in mainline. I dropped patches 15-19 because they are
in the way for your changes.

Thanks,

C.