[Qemu-devel] Multiple pci buses

[Qemu-devel] Multiple pci buses

[BALATON Zoltan]

Hello,

I'm trying to change hw/ppc/mac_newworld.c and hw/pci-host/uninorth.c to 
bring the mac99 machine type closer to what's seen in these dumps:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604134
http://nandra.segv.jp/NetBSD/G4.dump-device-tree.txt
http://raveland.org/ports/eeprom.txt
http://mail-index.netbsd.org/port-macppc/2007/10/24/0000.html

This machine seems to have 3 pci busses but the third is not important 
just the other two:

ff8721c0: /pci@f0000000
ff898cd0:   /uni-north-agp@b
ff898f40:   /ATY,Rage128Ps@10
ff873268: /pci@f2000000
ff8742d8:   /pci-bridge@d
ff876368:     /mac-io@7

The first is actually an AGP bus with the onboard graphics card and the 
second is where other devices are. It would be enough to create an empty 
bus without the AGP graphics card for now and have the devices on the 
second bus. The pci_pmac_init function in uninorth.c seems to have some 
commented out code to try to model this but that does not work. Can 
someone who understands this better please explain what should be done to 
model the above? How to create two pci buses and connect them to separate 
memory regions?

Thank you,
BALATON Zoltan

Re: [Qemu-devel] Multiple pci buses

[BALATON Zoltan]

On Sat, 15 Mar 2014, BALATON Zoltan wrote:
> I'm trying to change hw/ppc/mac_newworld.c and hw/pci-host/uninorth.c to 
> bring the mac99 machine type closer to what's seen in these dumps:

I've also tried the patch below based on what I've seen in pci_apb_init 
but it only results in this error:

qemu-system-ppc: hw/core/qdev.c:162: qdev_init: Assertion `!dev->realized' failed.
Aborted

Is there some explanation available somewhere on how to do this correctly?

Thank you,
BALATON Zoltan

diff --git a/hw/pci-host/uninorth.c b/hw/pci-host/uninorth.c
index e72fe2a..485c446 100644
--- a/hw/pci-host/uninorth.c
+++ b/hw/pci-host/uninorth.c
@@ -25,6 +25,8 @@
 #include "hw/ppc/mac.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_host.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_bus.h"

 /* debug UniNorth */
 //#define DEBUG_UNIN
@@ -214,54 +216,67 @@ static int pci_unin_internal_init_device(SysBusDevice *dev)

 PCIBus *pci_pmac_init(qemu_irq *pic,
                       MemoryRegion *address_space_mem,
-                      MemoryRegion *address_space_io)
+                      MemoryRegion *address_space_io,
+                      PCIBus **bus2)
 {
     DeviceState *dev;
     SysBusDevice *s;
     PCIHostState *h;
     UNINState *d;
+    PCIDevice *pdev;

     /* Use values found on a real PowerMac */
-    /* Uninorth main bus */
-    dev = qdev_create(NULL, TYPE_UNI_NORTH_PCI_HOST_BRIDGE);
+    /* Uninorth AGP bus */
+    dev = qdev_create(NULL, TYPE_UNI_NORTH_AGP_HOST_BRIDGE);
     qdev_init_nofail(dev);
     s = SYS_BUS_DEVICE(dev);
     h = PCI_HOST_BRIDGE(s);
-    d = UNI_NORTH_PCI_HOST_BRIDGE(dev);
+    d = UNI_NORTH_AGP_HOST_BRIDGE(dev);
+    /* config space */
+    sysbus_mmio_map(s, 0, 0xf1000000);
+    sysbus_mmio_map(s, 1, 0xf1400000);
+    /* mmio space */
+    memory_region_init(&d->pci_mmio, OBJECT(d), "agp-mmio", 0x100000000ULL);
+    memory_region_init_alias(&d->pci_hole, OBJECT(d), "agp-hole", &d->pci_mmio,
+                             0x90000000ULL, 0x10000000ULL);
+    memory_region_add_subregion(address_space_mem, 0x90000000ULL,
+                                &d->pci_hole);
+    h->bus = pci_register_bus(dev, NULL,
+                              pci_unin_set_irq, pci_unin_map_irq, pic,
+                              &d->pci_mmio,
+                              address_space_io,
+                              PCI_DEVFN(11, 0), 4, TYPE_PCI_BUS);
+    pci_create_simple(h->bus, PCI_DEVFN(11, 0), "uni-north-agp");
+
+    /* Uninorth PCI bus */
+    pdev = pci_create_simple_multifunction(h->bus, PCI_DEVFN(12, 0), true,
+                                           "uni-north-pci");
+    qdev_init_nofail(&pdev->qdev);
+    *bus2 = pci_bridge_get_sec_bus(PCI_BRIDGE(pdev));
+#if 0
+    d = UNI_NORTH_PCI_HOST_BRIDGE(pdev);
     memory_region_init(&d->pci_mmio, OBJECT(d), "pci-mmio", 0x100000000ULL);
     memory_region_init_alias(&d->pci_hole, OBJECT(d), "pci-hole", &d->pci_mmio,
-                             0x80000000ULL, 0x70000000ULL);
+                             0x80000000ULL, 0x10000000ULL);
     memory_region_add_subregion(address_space_mem, 0x80000000ULL,
                                 &d->pci_hole);

-    h->bus = pci_register_bus(dev, NULL,
+    bus2 = pci_register_bus(pdev, NULL,
                               pci_unin_set_irq, pci_unin_map_irq,
                               pic,
                               &d->pci_mmio,
-                              address_space_io,
+                              address_space_io + 0x2000000ULL,
                               PCI_DEVFN(11, 0), 4, TYPE_PCI_BUS);

-#if 0
-    pci_create_simple(h->bus, PCI_DEVFN(11, 0), "uni-north");
+    sysbus_mmio_map(s, 0, 0xf3000000);
+    sysbus_mmio_map(s, 1, 0xf3400000);
 #endif
-
-    sysbus_mmio_map(s, 0, 0xf2800000);
-    sysbus_mmio_map(s, 1, 0xf2c00000);
-
     /* DEC 21154 bridge */
 #if 0
     /* XXX: not activated as PPC BIOS doesn't handle multiple buses properly */
     pci_create_simple(h->bus, PCI_DEVFN(12, 0), "dec-21154");
 #endif

-    /* Uninorth AGP bus */
-    pci_create_simple(h->bus, PCI_DEVFN(11, 0), "uni-north-agp");
-    dev = qdev_create(NULL, TYPE_UNI_NORTH_AGP_HOST_BRIDGE);
-    qdev_init_nofail(dev);
-    s = SYS_BUS_DEVICE(dev);
-    sysbus_mmio_map(s, 0, 0xf0800000);
-    sysbus_mmio_map(s, 1, 0xf0c00000);
-
     /* Uninorth internal bus */
 #if 0
     /* XXX: not needed for now */
diff --git a/hw/ppc/mac.h b/hw/ppc/mac.h
index c1faf9c..c14c7d6 100644
--- a/hw/ppc/mac.h
+++ b/hw/ppc/mac.h
@@ -155,7 +155,8 @@ PCIBus *pci_grackle_init(uint32_t base, qemu_irq *pic,
 /* UniNorth PCI */
 PCIBus *pci_pmac_init(qemu_irq *pic,
                       MemoryRegion *address_space_mem,
-                      MemoryRegion *address_space_io);
+                      MemoryRegion *address_space_io,
+                      PCIBus **bus2);
 PCIBus *pci_pmac_u3_init(qemu_irq *pic,
                          MemoryRegion *address_space_mem,
                          MemoryRegion *address_space_io);
diff --git a/hw/ppc/mac_newworld.c b/hw/ppc/mac_newworld.c
index 5e79575..a074935 100644
--- a/hw/ppc/mac_newworld.c
+++ b/hw/ppc/mac_newworld.c
@@ -154,12 +154,14 @@ static void ppc_core99_init(QEMUMachineInitArgs *args)
     qemu_irq *pic, **openpic_irqs;
     MemoryRegion *isa = g_new(MemoryRegion, 1);
     MemoryRegion *unin_memory = g_new(MemoryRegion, 1);
+#if 0
     MemoryRegion *unin2_memory = g_new(MemoryRegion, 1);
+#endif
     int linux_boot, i, j, k;
     MemoryRegion *ram = g_new(MemoryRegion, 1), *bios = g_new(MemoryRegion, 1);
     hwaddr kernel_base, initrd_base, cmdline_base = 0;
     long kernel_size, initrd_size;
-    PCIBus *pci_bus;
+    PCIBus *pci_bus, *pci_bus2;
     PCIDevice *macio;
     MACIOIDEState *macio_ide;
     BusState *adb_bus;
@@ -296,10 +298,10 @@ static void ppc_core99_init(QEMUMachineInitArgs *args)
     /* UniN init: XXX should be a real device */
     memory_region_init_io(unin_memory, NULL, &unin_ops, token, "unin", 0x1000);
     memory_region_add_subregion(get_system_memory(), 0xf8000000, unin_memory);
-
+#if 0
     memory_region_init_io(unin2_memory, NULL, &unin_ops, token, "unin", 0x1000);
     memory_region_add_subregion(get_system_memory(), 0xf3000000, unin2_memory);
-
+#endif
     openpic_irqs = g_malloc0(smp_cpus * sizeof(qemu_irq *));
     openpic_irqs[0] =
         g_malloc0(smp_cpus * sizeof(qemu_irq) * OPENPIC_OUTPUT_NB);
@@ -367,11 +369,12 @@ static void ppc_core99_init(QEMUMachineInitArgs *args)
         pci_bus = pci_pmac_u3_init(pic, get_system_memory(), get_system_io());
         machine_arch = ARCH_MAC99_U3;
     } else {
-        pci_bus = pci_pmac_init(pic, get_system_memory(), get_system_io());
+        pci_bus = pci_pmac_init(pic, get_system_memory(), get_system_io(),
+                                &pci_bus2);
         machine_arch = ARCH_MAC99;
     }
     /* init basic PC hardware */
-    pci_vga_init(pci_bus);
+    pci_vga_init(pci_bus2);

     escc_mem = escc_init(0, pic[0x25], pic[0x24],
                          serial_hds[0], serial_hds[1], ESCC_CLOCK, 4);

Re: [Qemu-devel] Multiple pci buses

[Alexey Kardashevskiy]

On 03/16/2014 08:21 AM, BALATON Zoltan wrote:
> Hello,
> 
> I'm trying to change hw/ppc/mac_newworld.c and hw/pci-host/uninorth.c to
> bring the mac99 machine type closer to what's seen in these dumps:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604134
> http://nandra.segv.jp/NetBSD/G4.dump-device-tree.txt
> http://raveland.org/ports/eeprom.txt
> http://mail-index.netbsd.org/port-macppc/2007/10/24/0000.html
> 
> This machine seems to have 3 pci busses but the third is not important just
> the other two:
> 
> ff8721c0: /pci@f0000000
> ff898cd0:   /uni-north-agp@b
> ff898f40:   /ATY,Rage128Ps@10
> ff873268: /pci@f2000000
> ff8742d8:   /pci-bridge@d
> ff876368:     /mac-io@7
> 
> The first is actually an AGP bus with the onboard graphics card and the
> second is where other devices are. It would be enough to create an empty
> bus without the AGP graphics card for now and have the devices on the
> second bus. The pci_pmac_init function in uninorth.c seems to have some
> commented out code to try to model this but that does not work. Can someone
> who understands this better please explain what should be done to model the
> above? How to create two pci buses and connect them to separate memory
> regions?

[adding qemu-devel@nongnu.org to cc]

I have no idea how that "pmac" works... On pseries you can add multiple
PHBs via "-device spapr-pci-host-bridge,id=lalala" and then put devices
onto it by "-device e1000,bus=lalala.0" so this should be good reference to
start with.




-- 
Alexey

Re: [Qemu-devel] Multiple pci buses

[Andreas Färber]

Hi,

Am 15.03.2014 22:21, schrieb BALATON Zoltan:
> I'm trying to change hw/ppc/mac_newworld.c and hw/pci-host/uninorth.c to
> bring the mac99 machine type closer to what's seen in these dumps:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604134
> http://nandra.segv.jp/NetBSD/G4.dump-device-tree.txt
> http://raveland.org/ports/eeprom.txt
> http://mail-index.netbsd.org/port-macppc/2007/10/24/0000.html
> 
> This machine seems to have 3 pci busses but the third is not important
> just the other two:
> 
> ff8721c0: /pci@f0000000
> ff898cd0:   /uni-north-agp@b
> ff898f40:   /ATY,Rage128Ps@10
> ff873268: /pci@f2000000
> ff8742d8:   /pci-bridge@d
> ff876368:     /mac-io@7
> 
> The first is actually an AGP bus with the onboard graphics card and the
> second is where other devices are. It would be enough to create an empty
> bus without the AGP graphics card for now and have the devices on the
> second bus. The pci_pmac_init function in uninorth.c seems to have some
> commented out code to try to model this but that does not work. Can
> someone who understands this better please explain what should be done
> to model the above? How to create two pci buses and connect them to
> separate memory regions?

In earlier times QEMU did not properly support multiple PCI bus domains.
Some code in
http://git.qemu.org/?p=qemu.git;a=blob;f=hw/pci-host/uninorth.c;h=e72fe2a70b954bf5675ad0c8735fea6bad665be6;hb=HEAD
is #if 0'ed out that you should take a look at.

I had investigated that some time ago based on a G4 in our office and
might be able to revive some patches... Please keep me CC'ed.

Cheers,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] Multiple pci buses

[BALATON Zoltan]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2166 bytes --]

On Mon, 17 Mar 2014, Andreas Färber wrote:
> In earlier times QEMU did not properly support multiple PCI bus domains.
> Some code in
> http://git.qemu.org/?p=qemu.git;a=blob;f=hw/pci-host/uninorth.c;h=e72fe2a70b954bf5675ad0c8735fea6bad665be6;hb=HEAD
> is #if 0'ed out that you should take a look at.

I've seen these #if 0'ed parts but just enabling them does not seem to be 
enough. I don't know enough about how should all this work and found no 
documentation or examples to follow so I hope someone can explain what it 
takes to create two pci buses (so devices added to the first get 
0:dev:func and those added to the second get 1:dev:func addresses) and 
these busses have their Cfg/IO/MMIO space mapped to different addresses. 
The patch I came up with so far did not work. The pci buses and memory map 
from the dumps I've seen should look like this:

0000:00:0b.0 Host bridge [0600]: Apple Computer Inc. UniNorth AGP [106b:0020]
0000:00:10.0 VGA compatible controller [0300]: ATI Technologies Inc Radeon R200 QL [Radeon 8500 LE] [1002:514c]

0001:10:0b.0 Host bridge [0600]: Apple Computer Inc. UniNorth PCI [106b:001f]
0001:10:0d.0 PCI bridge [0604]: Digital Equipment Corporation DECchip 21154 [1011:0026] (rev 05)
0001:11:07.0 Unassigned class [ff00]: Apple Computer Inc. KeyLargo Mac I/O [106b:0022] (rev 02)

corresponding to this openfirmware tree:

ff8721c0: /pci@f0000000
ff898cd0:   /uni-north-agp@b
ff898f40:   /ATY,Rage128Ps@10
ff873268: /pci@f2000000
ff8742d8:   /pci-bridge@d
ff876368:     /mac-io@7

and the memory mapping is:

80000000-8fffffff : /pci@f2000000
    80000000-800fffff : PCI Bus 0001:11
      80000000-8007ffff : 0001:11:07.0
        80000000-8007ffff : 0.80000000:mac-io

90000000-9fffffff : /pci@f0000000
    90000000-9000ffff : 0000:00:10.0
      90000000-9000ffff : radeonfb mmio

f1000000-f1ffffff : /pci@f0000000
f3000000-f3ffffff : /pci@f2000000

> I had investigated that some time ago based on a G4 in our office and
> might be able to revive some patches... Please keep me CC'ed.

If you have any info/patches for this they are very welcome.

Regards,
BALATON Zoltan

Re: [Qemu-devel] Multiple pci buses

[Andreas Färber]

Am 17.03.2014 23:14, schrieb BALATON Zoltan:
> On Mon, 17 Mar 2014, Andreas Färber wrote:
>> In earlier times QEMU did not properly support multiple PCI bus domains.
>> Some code in
>> http://git.qemu.org/?p=qemu.git;a=blob;f=hw/pci-host/uninorth.c;h=e72fe2a70b954bf5675ad0c8735fea6bad665be6;hb=HEAD
>>
>> is #if 0'ed out that you should take a look at.
> 
> I've seen these #if 0'ed parts but just enabling them does not seem to
> be enough. I don't know enough about how should all this work and found
> no documentation or examples to follow so I hope someone can explain
> what it takes to create two pci buses (so devices added to the first get
> 0:dev:func and those added to the second get 1:dev:func addresses) and
> these busses have their Cfg/IO/MMIO space mapped to different addresses.
> The patch I came up with so far did not work. The pci buses and memory
> map from the dumps I've seen should look like this:
> 
> 0000:00:0b.0 Host bridge [0600]: Apple Computer Inc. UniNorth AGP
> [106b:0020]
> 0000:00:10.0 VGA compatible controller [0300]: ATI Technologies Inc
> Radeon R200 QL [Radeon 8500 LE] [1002:514c]
> 
> 0001:10:0b.0 Host bridge [0600]: Apple Computer Inc. UniNorth PCI
> [106b:001f]
> 0001:10:0d.0 PCI bridge [0604]: Digital Equipment Corporation DECchip
> 21154 [1011:0026] (rev 05)
> 0001:11:07.0 Unassigned class [ff00]: Apple Computer Inc. KeyLargo Mac
> I/O [106b:0022] (rev 02)
> 
> corresponding to this openfirmware tree:
> 
> ff8721c0: /pci@f0000000
> ff898cd0:   /uni-north-agp@b
> ff898f40:   /ATY,Rage128Ps@10
> ff873268: /pci@f2000000
> ff8742d8:   /pci-bridge@d
> ff876368:     /mac-io@7
> 
> and the memory mapping is:
> 
> 80000000-8fffffff : /pci@f2000000
>    80000000-800fffff : PCI Bus 0001:11
>      80000000-8007ffff : 0001:11:07.0
>        80000000-8007ffff : 0.80000000:mac-io
> 
> 90000000-9fffffff : /pci@f0000000
>    90000000-9000ffff : 0000:00:10.0
>      90000000-9000ffff : radeonfb mmio
> 
> f1000000-f1ffffff : /pci@f0000000
> f3000000-f3ffffff : /pci@f2000000
> 
>> I had investigated that some time ago based on a G4 in our office and
>> might be able to revive some patches... Please keep me CC'ed.
> 
> If you have any info/patches for this they are very welcome.

Just stumbled over a text file of our G4:

--->8---

0000:00:0b.0 Apple Computer Inc. UniNorth AGP [106b:0020]

0001:10:0b.0 Apple Computer Inc. UniNorth PCI [106b:001f]
0001:10:0d.0 Digital Equipment Corporation DECchip 21154 [1011:0026]

0002:21:0b.0 Apple Computer Inc. UniNorth Internal PCI [106b:001e]

--->8---

The first three match yours, but I have an additional internal PCI.

Cheers,
Andreas

P.S. Forgot to mention, do not add #if 0s in patches. Either keep code
compiling or drop it properly. Otherwise it will bit-rot.

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] Multiple pci buses

[BALATON Zoltan]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1083 bytes --]

On Wed, 19 Mar 2014, Andreas Färber wrote:
> Just stumbled over a text file of our G4:
>
> --->8---
>
> 0000:00:0b.0 Apple Computer Inc. UniNorth AGP [106b:0020]
>
> 0001:10:0b.0 Apple Computer Inc. UniNorth PCI [106b:001f]
> 0001:10:0d.0 Digital Equipment Corporation DECchip 21154 [1011:0026]
>
> 0002:21:0b.0 Apple Computer Inc. UniNorth Internal PCI [106b:001e]
>
> --->8---
>
> The first three match yours, but I have an additional internal PCI.

The third bus is also in the dumps I've referred to with an ethernet 
device connected to it, but I think it is not important to model it now so 
I did not include it in the question for brevity. I'd be happy if I could 
implement the other two for now but I still don't know how to do that.

> P.S. Forgot to mention, do not add #if 0s in patches. Either keep code
> compiling or drop it properly. Otherwise it will bit-rot.

The patch I've sent was not final by any means, it was only to show what 
I've tried and what I was asking about. I'll clean up in the final 
version.

Regards,
BALATON Zoltan