* How to use pxb-pcie in correct way?
@ 2024-04-08 11:58 Marcin Juszkiewicz
2024-04-09 15:14 ` Jonathan Cameron via
0 siblings, 1 reply; 2+ messages in thread
From: Marcin Juszkiewicz @ 2024-04-08 11:58 UTC (permalink / raw)
To: qemu-devel
Cc: Michael S. Tsirkin, Marcel Apfelbaum, Leif Lindholm,
Ard Biesheuvel, Xiong Yining, Chen Baozi
For quite a while I am experimenting with PCI Express setup on SBSA-Ref
system. And finally decided to write.
We want to play with NUMA setup and "pxb-pcie" can be assigned to NUMA
node other than cpu0 one. But adding it makes other cards dissapear...
When I boot sbsa-ref I have plain PCIe setup:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81084000 [0x81084fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Adding extra PCIe card works fine - both just "igb" and "igb" with
"pcie-root-port".
But adding "pcie-root-port" + "igb" and then "pxb-pcie" makes "igb"
dissapear:
../code/qemu/build/qemu-system-aarch64
-monitor telnet::45454,server,nowait
-serial stdio
-device pcie-root-port,id=ULyWl,slot=0,chassis=0
-device igb,bus=ULyWl
-device pxb-pcie,bus_nr=1
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81085000 [0x81085fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 1.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81084000 [0x81084fff].
id "ULyWl"
Bus 0, device 4, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
If I add "igb" directly (without root port) then it appears correctly:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x810c4000 [0x810c4fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
Ethernet controller: PCI device 8086:10c9
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
id ""
Bus 0, device 4, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
When I add "pcie-root-port" with "igb" followed by "pcie-root-port" and
"pxb-pcie" then no IGB again:
-device pcie-root-port,id=RjKXs,slot=0,chassis=0
-device igb,bus=RjKXs
-device pcie-root-port,chassis=7
-device pxb-pcie,bus_nr=1
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81086000 [0x81086fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 1.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81085000 [0x81085fff].
id "RjKXs"
Bus 0, device 4, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 2.
subordinate bus 2.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81084000 [0x81084fff].
id ""
Bus 0, device 5, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
Same (no "igb") with more complex structure behind "pxb-pcie" bridge:
../code/qemu/build/qemu-system-aarch64
-monitor telnet::45454,server,nowait
-serial stdio
-device pcie-root-port,id=PrZUz,slot=0,chassis=0
-device igb,bus=PrZUz
-device pcie-root-port,chassis=7
-device pxb-pcie,id=pxb1,bus_nr=1
-device pcie-pci-bridge,id=pci,bus=pxb1
-device es1370,bus=pci,addr=9,id=es1370
-device e1000,bus=pci,addr=10,id=e1000
-device pcie-root-port,id=root_port_for_ahci,bus=pxb1,chassis=10,slot=0
-device ahci,bus=root_port_for_ahci,id=ahci
Bus 1, device 0, function 0:
PCI bridge: PCI device 1b36:000e
IRQ 255, pin A
BUS 1.
secondary bus 2.
subordinate bus 2.
IO range [0x1000, 0x1fff]
memory range [0x81100000, 0x811fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 64 bit memory at 0x81200000 [0x812000ff].
id "pci"
Bus 2, device 9, function 0:
Audio controller: PCI device 1274:5000
PCI subsystem 4942:4c4c
IRQ 255, pin A
BAR0: I/O at 0xffffffffffffffff [0x00fe].
id "es1370"
Bus 2, device 16, function 0:
Ethernet controller: PCI device 8086:100e
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: I/O at 0xffffffffffffffff [0x003e].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id "e1000"
Bus 1, device 1, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 1.
secondary bus 3.
subordinate bus 3.
IO range [0x0000, 0x0fff]
memory range [0x81000000, 0x810fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81201000 [0x81201fff].
id "root_port_for_ahci"
Bus 3, device 0, function 0:
SATA controller: PCI device 8086:2922
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR4: I/O at 0xffffffffffffffff [0x001e].
BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
id "ahci"
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81386000 [0x81386fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 3.
IO range [0x0000, 0x1fff]
memory range [0x81000000, 0x812fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81385000 [0x81385fff].
id "PrZUz"
Bus 1, device 0, function 0:
PCI bridge: PCI device 1b36:000e
IRQ 255, pin A
BUS 1.
secondary bus 2.
subordinate bus 2.
IO range [0x1000, 0x1fff]
memory range [0x81100000, 0x811fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 64 bit memory at 0x81200000 [0x812000ff].
id "pci"
Bus 2, device 9, function 0:
Audio controller: PCI device 1274:5000
PCI subsystem 4942:4c4c
IRQ 255, pin A
BAR0: I/O at 0xffffffffffffffff [0x00fe].
id "es1370"
Bus 2, device 16, function 0:
Ethernet controller: PCI device 8086:100e
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: I/O at 0xffffffffffffffff [0x003e].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id "e1000"
Bus 1, device 1, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 1.
secondary bus 3.
subordinate bus 3.
IO range [0x0000, 0x0fff]
memory range [0x81000000, 0x810fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81201000 [0x81201fff].
id "root_port_for_ahci"
Bus 3, device 0, function 0:
SATA controller: PCI device 8086:2922
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR4: I/O at 0xffffffffffffffff [0x001e].
BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
id "ahci"
Bus 0, device 4, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 4.
subordinate bus 4.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81384000 [0x81384fff].
id ""
Bus 0, device 5, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id "pxb1"
What happens with missing card? It does not have to be "igb" - "ac97"
gets lost too.
And only first added one dissapears. If I add two cards (with root-port
entries) before "pxb-pcie" then second one is present.
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: How to use pxb-pcie in correct way?
2024-04-08 11:58 How to use pxb-pcie in correct way? Marcin Juszkiewicz
@ 2024-04-09 15:14 ` Jonathan Cameron via
0 siblings, 0 replies; 2+ messages in thread
From: Jonathan Cameron via @ 2024-04-09 15:14 UTC (permalink / raw)
To: Marcin Juszkiewicz
Cc: qemu-devel, Michael S. Tsirkin, Marcel Apfelbaum, Leif Lindholm,
Ard Biesheuvel, Xiong Yining, Chen Baozi
On Mon, 8 Apr 2024 13:58:00 +0200
Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> wrote:
> For quite a while I am experimenting with PCI Express setup on SBSA-Ref
> system. And finally decided to write.
>
> We want to play with NUMA setup and "pxb-pcie" can be assigned to NUMA
> node other than cpu0 one. But adding it makes other cards dissapear...
>
> When I boot sbsa-ref I have plain PCIe setup:
>
> (qemu) info pci
> Bus 0, device 0, function 0:
> Host bridge: PCI device 1b36:0008
> PCI subsystem 1af4:1100
> id ""
> Bus 0, device 1, function 0:
> Ethernet controller: PCI device 8086:10d3
> PCI subsystem 8086:0000
> IRQ 255, pin A
> BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
> BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
> BAR2: I/O at 0xffffffffffffffff [0x001e].
> BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
> BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
> id ""
> Bus 0, device 2, function 0:
> Display controller: PCI device 1234:1111
> PCI subsystem 1af4:1100
> BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
> BAR2: 32 bit memory at 0x81084000 [0x81084fff].
> BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
> id ""
>
> Adding extra PCIe card works fine - both just "igb" and "igb" with
> "pcie-root-port".
>
> But adding "pcie-root-port" + "igb" and then "pxb-pcie" makes "igb"
> dissapear:
>
> ../code/qemu/build/qemu-system-aarch64
> -monitor telnet::45454,server,nowait
> -serial stdio
> -device pcie-root-port,id=ULyWl,slot=0,chassis=0
> -device igb,bus=ULyWl
> -device pxb-pcie,bus_nr=1
That's setting the base bus number to 1. Very likely to clash with the bus
number for the bus below the root port.
Set it to bu_nr=128 or something like that.
There is no sanity checking for PXBs because the bus enumeration is
an EDK2 problem in general - short of enumerating the buses in QEMU
there isn't a way for it to tell.
J
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2024-04-09 15:15 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-08 11:58 How to use pxb-pcie in correct way? Marcin Juszkiewicz
2024-04-09 15:14 ` Jonathan Cameron via
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).