From: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
To: qemu-devel@nongnu.org
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
Marcel Apfelbaum <marcel.apfelbaum@gmail.com>,
Leif Lindholm <quic_llindhol@quicinc.com>,
Ard Biesheuvel <ardb+tianocore@kernel.org>,
Xiong Yining <xiongyining1480@phytium.com.cn>,
Chen Baozi <chenbaozi@phytium.com.cn>
Subject: How to use pxb-pcie in correct way?
Date: Mon, 8 Apr 2024 13:58:00 +0200 [thread overview]
Message-ID: <8b9b96e1-faaa-4866-aeb9-c439d5f83139@linaro.org> (raw)
For quite a while I am experimenting with PCI Express setup on SBSA-Ref
system. And finally decided to write.
We want to play with NUMA setup and "pxb-pcie" can be assigned to NUMA
node other than cpu0 one. But adding it makes other cards dissapear...
When I boot sbsa-ref I have plain PCIe setup:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81084000 [0x81084fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Adding extra PCIe card works fine - both just "igb" and "igb" with
"pcie-root-port".
But adding "pcie-root-port" + "igb" and then "pxb-pcie" makes "igb"
dissapear:
../code/qemu/build/qemu-system-aarch64
-monitor telnet::45454,server,nowait
-serial stdio
-device pcie-root-port,id=ULyWl,slot=0,chassis=0
-device igb,bus=ULyWl
-device pxb-pcie,bus_nr=1
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81085000 [0x81085fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 1.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81084000 [0x81084fff].
id "ULyWl"
Bus 0, device 4, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
If I add "igb" directly (without root port) then it appears correctly:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x810c4000 [0x810c4fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
Ethernet controller: PCI device 8086:10c9
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
id ""
Bus 0, device 4, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
When I add "pcie-root-port" with "igb" followed by "pcie-root-port" and
"pxb-pcie" then no IGB again:
-device pcie-root-port,id=RjKXs,slot=0,chassis=0
-device igb,bus=RjKXs
-device pcie-root-port,chassis=7
-device pxb-pcie,bus_nr=1
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81086000 [0x81086fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 1.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81085000 [0x81085fff].
id "RjKXs"
Bus 0, device 4, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 2.
subordinate bus 2.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81084000 [0x81084fff].
id ""
Bus 0, device 5, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id ""
Same (no "igb") with more complex structure behind "pxb-pcie" bridge:
../code/qemu/build/qemu-system-aarch64
-monitor telnet::45454,server,nowait
-serial stdio
-device pcie-root-port,id=PrZUz,slot=0,chassis=0
-device igb,bus=PrZUz
-device pcie-root-port,chassis=7
-device pxb-pcie,id=pxb1,bus_nr=1
-device pcie-pci-bridge,id=pci,bus=pxb1
-device es1370,bus=pci,addr=9,id=es1370
-device e1000,bus=pci,addr=10,id=e1000
-device pcie-root-port,id=root_port_for_ahci,bus=pxb1,chassis=10,slot=0
-device ahci,bus=root_port_for_ahci,id=ahci
Bus 1, device 0, function 0:
PCI bridge: PCI device 1b36:000e
IRQ 255, pin A
BUS 1.
secondary bus 2.
subordinate bus 2.
IO range [0x1000, 0x1fff]
memory range [0x81100000, 0x811fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 64 bit memory at 0x81200000 [0x812000ff].
id "pci"
Bus 2, device 9, function 0:
Audio controller: PCI device 1274:5000
PCI subsystem 4942:4c4c
IRQ 255, pin A
BAR0: I/O at 0xffffffffffffffff [0x00fe].
id "es1370"
Bus 2, device 16, function 0:
Ethernet controller: PCI device 8086:100e
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: I/O at 0xffffffffffffffff [0x003e].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id "e1000"
Bus 1, device 1, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 1.
secondary bus 3.
subordinate bus 3.
IO range [0x0000, 0x0fff]
memory range [0x81000000, 0x810fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81201000 [0x81201fff].
id "root_port_for_ahci"
Bus 3, device 0, function 0:
SATA controller: PCI device 8086:2922
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR4: I/O at 0xffffffffffffffff [0x001e].
BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
id "ahci"
Bus 0, device 0, function 0:
Host bridge: PCI device 1b36:0008
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
Ethernet controller: PCI device 8086:10d3
PCI subsystem 8086:0000
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR2: I/O at 0xffffffffffffffff [0x001e].
BAR3: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id ""
Bus 0, device 2, function 0:
Display controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0x80000000 [0x80ffffff].
BAR2: 32 bit memory at 0x81386000 [0x81386fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x00007ffe].
id ""
Bus 0, device 3, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 1.
subordinate bus 3.
IO range [0x0000, 0x1fff]
memory range [0x81000000, 0x812fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81385000 [0x81385fff].
id "PrZUz"
Bus 1, device 0, function 0:
PCI bridge: PCI device 1b36:000e
IRQ 255, pin A
BUS 1.
secondary bus 2.
subordinate bus 2.
IO range [0x1000, 0x1fff]
memory range [0x81100000, 0x811fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 64 bit memory at 0x81200000 [0x812000ff].
id "pci"
Bus 2, device 9, function 0:
Audio controller: PCI device 1274:5000
PCI subsystem 4942:4c4c
IRQ 255, pin A
BAR0: I/O at 0xffffffffffffffff [0x00fe].
id "es1370"
Bus 2, device 16, function 0:
Ethernet controller: PCI device 8086:100e
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
BAR1: I/O at 0xffffffffffffffff [0x003e].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
id "e1000"
Bus 1, device 1, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 1.
secondary bus 3.
subordinate bus 3.
IO range [0x0000, 0x0fff]
memory range [0x81000000, 0x810fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81201000 [0x81201fff].
id "root_port_for_ahci"
Bus 3, device 0, function 0:
SATA controller: PCI device 8086:2922
PCI subsystem 1af4:1100
IRQ 255, pin A
BAR4: I/O at 0xffffffffffffffff [0x001e].
BAR5: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
id "ahci"
Bus 0, device 4, function 0:
PCI bridge: PCI device 1b36:000c
IRQ 255, pin A
BUS 0.
secondary bus 4.
subordinate bus 4.
IO range [0xf000, 0x0fff]
memory range [0xfff00000, 0x000fffff]
prefetchable memory range [0xfffffffffff00000, 0x000fffff]
BAR0: 32 bit memory at 0x81384000 [0x81384fff].
id ""
Bus 0, device 5, function 0:
Host bridge: PCI device 1b36:000b
PCI subsystem 1af4:1100
id "pxb1"
What happens with missing card? It does not have to be "igb" - "ac97"
gets lost too.
And only first added one dissapears. If I add two cards (with root-port
entries) before "pxb-pcie" then second one is present.
next reply other threads:[~2024-04-08 11:58 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-08 11:58 Marcin Juszkiewicz [this message]
2024-04-09 15:14 ` How to use pxb-pcie in correct way? Jonathan Cameron via
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8b9b96e1-faaa-4866-aeb9-c439d5f83139@linaro.org \
--to=marcin.juszkiewicz@linaro.org \
--cc=ardb+tianocore@kernel.org \
--cc=chenbaozi@phytium.com.cn \
--cc=marcel.apfelbaum@gmail.com \
--cc=mst@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quic_llindhol@quicinc.com \
--cc=xiongyining1480@phytium.com.cn \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).