[PATCH v1 00/11] Support PCIe RC to AST2600 and AST2700

[Jamin Lin via <qemu-arm@nongnu.org>]

v1:
 1. Add PCIe PHY, CFG, and MMIO window support for AST2600.
    Note: Only supports RC_H.
 2. Add PCIe PHY, CFG, and MMIO window support for AST2700.
    Note: Supports 3 RCs.

Testing PCIe RC model with e1000e PCIe device model

AST2600

The AST2600/ASPEED PCIe driver treats root bus 0x80 specially:
Only two device addresses are usable on the root bus:
addr 0: ASPEED host bridge
addr 8: usable for attaching a PCIe root port

To attach endpoints without driver changes, the test places a QEMU PCIe root
port at 0x80:08.0 and attaches an e1000e NIC behind it.
The endpoint then enumerates on bus 0x81.

QEMU command line additions
-device pcie-root-port,id=root_port0,slot=1,addr=8,bus=pcie.0
-device e1000e,netdev=net0,bus=root_port0
-netdev user,id=net0

Verification with lspci:
0001:80:00.0 Host bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
0001:80:08.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
0001:81:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection

This is a temporary solution that allows attaching multiple PCIe
devices while the ASPEED drivers does not support placing endpoints directly
on bus numbers 0x80.

Reference:
https://github.com/AspeedTech-BMC/linux/blob/aspeed-master-v6.6/drivers/pci/controller/pcie-aspeed.c#L309

Test Image: ASPEED SDK v09.07. It okay to use ASPEED SDK v09.06.
https://github.com/AspeedTech-BMC/openbmc/releases/download/v09.07/ast2600-default-obmc.tar.gz

The e1000e driver is built into this image.
After booting, users should see the e1000e Ethernet interface.
With this setup, RC_H on AST2600 can be tested using e1000e endpoints in QEMU.

Example system output:
root@ast2600-default:~# lspci
80:00.0 Host bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
80:08.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
81:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
root@ast2600-default:~# ifconfig eth4
eth4      Link encap:Ethernet  HWaddr 52:54:00:12:34:5A
          inet addr:169.254.160.167  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5054:ff:fe12:345a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4896 (4.7 KiB)  TX bytes:8079 (7.8 KiB)
          Interrupt:81 Memory:70040000-70060000

root@ast2600-default:~# dmesg | grep "e1000e"
[    2.901955] e1000e: Intel(R) PRO/1000 Network Driver
[    2.902346] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[   15.366727] e1000e 0000:81:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   15.431884] e1000e 0000:81:00.0 0000:81:00.0 (uninitialized): registered PHC clock
[   15.508477] e1000e 0000:81:00.0 eth4: (PCI Express:2.5GT/s:Width x1) 52:54:00:12:34:5a
[   15.509301] e1000e 0000:81:00.0 eth4: Intel(R) PRO/1000 Network Connection
[   15.510768] e1000e 0000:81:00.0 eth4: MAC: 3, PHY: 8, PBA No: 000000-000
[   49.781217] e1000e 0000:81:00.0 eth4: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx


AST2700:

The ASPEED PCIe driver only supports bus 0 with a single device at slot 0,
which is reserved for the internal bridge. All endpoint devices must therefore
reside on bus 1 to be discovered.
See the ASPEED PCIe driver implementation here:
https://github.com/AspeedTech-BMC/linux/blob/aspeed-master-v6.6/drivers/pci/controller/pcie-aspeed.c#L512

By default, the ASPEED SDK enables only PCIe RC2, so initial testing covers RC2 only.

To attach e1000e devices and test all three RCs, I created a new test image
with driver modifications. This is a temporary solution. A better approach is
still needed to allow placing e1000e directly under bus 1.

Test release: https://github.com/jamin-aspeed/openbmc/releases/tag/qemu-test-0907
Image download: https://github.com/jamin-aspeed/openbmc/releases/download/qemu-test-0907/ast2700-default-pcie-qemu.tar.xz
This image includes the following changes:
Kernel patch (AST2700 RC mode + PCIe driver updates):
https://github.com/jamin-aspeed/openbmc/releases/download/qemu-test-0907/0001-PCI-aspeed-Enable-AST2700-RC-mode-support-and-adjust.patch

OpenBMC patch (installs e1000e modules into rootfs for testing):
https://github.com/jamin-aspeed/openbmc/releases/download/qemu-test-0907/0001-packagegroup-aspeed-Add-Intel-e1000-e1000e-kernel-mo.patch

QEMU Test Command

Each PCIe RC bus adds a pcie-root-port with an e1000e NIC behind it:

 -device pcie-root-port,id=root_port0,multifunction=on,slot=0,addr=1,bus=pcie.0 \
 -device e1000e,netdev=net0,bus=root_port0 \
 -netdev user,id=net0,hostfwd=:127.0.0.1:3222-:22,hostname=qemu0 \
 -device pcie-root-port,id=root_port1,slot=1,addr=2,bus=pcie.1 \
 -device e1000e,netdev=net1,bus=root_port1 \
 -netdev user,id=net1,hostfwd=:127.0.0.1:4222-:22,hostname=qemu1 \
 -device pcie-root-port,id=root_port2,slot=2,addr=3,bus=pcie.2 \
 -device e1000e,netdev=net2,bus=root_port2 \
 -netdev user,id=net2,hostfwd=:127.0.0.1:5222-:22,hostname=qemu2 \

Expected Results

With lspci, each RC should show the internal bridge, root port, and e1000e endpoint:

root@ast2700-default:~# lspci
0000:00:00.0 Host bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
0000:00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
0000:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0001:00:00.0 Host bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
0001:00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
0001:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0002:00:00.0 Host bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
0002:00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
0002:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Load the e1000e driver:
Kernel logs confirm device initialization and link-up at 1 Gbps Full Duplex on eth2/eth3/eth4.
root@ast2700-default:~# modprobe e1000e
[   59.181623] e1000e: Intel(R) PRO/1000 Network Driver
[   59.181926] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[   59.183508] e1000e 0000:01:00.0: enabling device (0000 -> 0002)
[   59.188808] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   59.252590] e1000e 0000:01:00.0 0000:01:00.0 (uninitialized): registered PHC clock
[   59.330877] e1000e 0000:01:00.0 eth2: (PCI Express:2.5GT/s:Width x1) 52:54:00:12:34:56
[   59.331990] e1000e 0000:01:00.0 eth2: Intel(R) PRO/1000 Network Connection
[   59.333211] e1000e 0000:01:00.0 eth2: MAC: 3, PHY: 8, PBA No: 000000-000
[   59.334881] e1000e 0001:01:00.0: enabling device (0000 -> 0002)
[   59.341616] e1000e 0001:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   59.394712] e1000e 0001:01:00.0 0001:01:00.0 (uninitialized): registered PHC clock
[   59.456674] e1000e 0001:01:00.0 eth3: (PCI Express:2.5GT/s:Width x1) 52:54:00:12:34:57
[   59.457487] e1000e 0001:01:00.0 eth3: Intel(R) PRO/1000 Network Connection
[   59.458033] e1000e 0001:01:00.0 eth3: MAC: 3, PHY: 8, PBA No: 000000-000
[   59.460037] e1000e 0002:01:00.0: enabling device (0000 -> 0002)
[   59.466403] e1000e 0002:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   59.533361] e1000e 0002:01:00.0 0002:01:00.0 (uninitialized): registered PHC clock
[   59.654491] 8021q: adding VLAN 0 to HW filter on device eth2
[   59.662100] e1000e 0002:01:00.0 eth4: (PCI Express:2.5GT/s:Width x1) 52:54:00:12:34:58
[   59.662966] e1000e 0002:01:00.0 eth4: Intel(R) PRO/1000 Network Connection
[   59.663527] e1000e 0002:01:00.0 eth4: MAC: 3, PHY: 8, PBA No: 000000-000
root@ast2700-default:~# [   59.814984] 8021q: adding VLAN 0 to HW filter on device eth3
[   59.938859] 8021q: adding VLAN 0 to HW filter on device eth4
[   59.954690] e1000e 0000:01:00.0 eth2: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[   60.116790] e1000e 0001:01:00.0 eth3: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[   60.237622] e1000e 0002:01:00.0 eth4: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

After loading, users should see:
eth2: PCIe0
eth3: PCIe1
eth4: PCIe2

Example ifconfig output shows all interfaces (eth0-eth4) active, with eth2-eth4 bound to the e1000e NICs.
With this setup, all three PCIe RCs on AST2700 can be tested using e1000e endpoints in QEMU.

root@ast2700-default:~# ifconfig
eth2      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
          inet addr:169.254.120.135  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:51 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5227 (5.1 KiB)  TX bytes:8551 (8.3 KiB)
          Interrupt:42 Memory:60040000-60060000

eth3      Link encap:Ethernet  HWaddr 52:54:00:12:34:57
          inet addr:169.254.237.52  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5054:ff:fe12:3457/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4786 (4.6 KiB)  TX bytes:8175 (7.9 KiB)
          Interrupt:45 Memory:80040000-80060000

eth4      Link encap:Ethernet  HWaddr 52:54:00:12:34:58
          inet addr:169.254.84.173  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5054:ff:fe12:3458/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4786 (4.6 KiB)  TX bytes:8191 (7.9 KiB)
          Interrupt:48 Memory:a0040000-a0060000

Jamin Lin (11):
  hw/pci/pci_ids Add PCI vendor ID for ASPEED
  hw/pci-host/aspeed: Add AST2600 PCIe PHY model
  hw/pci-host/aspeed: Add AST2600 PCIe config and host bridge
  hw/pci-host/aspeed: Add MSI support and per-RC IOMMU address space
  hw/arm/aspeed: Wire up PCIe devices in SoC model
  hw/arm/aspeed_ast2600: Add PCIe RC support (RC_H only)
  tests/functional/test_arm_aspeed_ast2600: Add PCIe test via root port
    and e1000e
  hw/pci-host/aspeed: Add AST2700 PCIe PHY
  hw/pci-host/aspeed: Add AST2700 PCIe config with dedicated H2X blocks
  hw/arm/aspeed_ast27x0: Introduce 3 PCIe RCs for AST2700
  tests/functional: Add PCIe presence test for AST2700

 include/hw/arm/aspeed_soc.h                   |  14 +
 include/hw/pci-host/aspeed_pcie.h             | 125 +++
 include/hw/pci/pci_ids.h                      |   2 +
 hw/arm/aspeed_ast2600.c                       |  69 +-
 hw/arm/aspeed_ast27x0.c                       |  61 ++
 hw/pci-host/aspeed_pcie.c                     | 955 ++++++++++++++++++
 hw/arm/Kconfig                                |   3 +
 hw/pci-host/Kconfig                           |   4 +
 hw/pci-host/meson.build                       |   1 +
 hw/pci-host/trace-events                      |  11 +
 .../functional/test_aarch64_aspeed_ast2700.py |   7 +
 .../test_aarch64_aspeed_ast2700fc.py          |   6 +
 tests/functional/test_arm_aspeed_ast2600.py   |  14 +
 13 files changed, 1269 insertions(+), 3 deletions(-)
 create mode 100644 include/hw/pci-host/aspeed_pcie.h
 create mode 100644 hw/pci-host/aspeed_pcie.c

-- 
2.43.0