From: Alex Williamson <alex.williamson@redhat.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
iommu@lists.linux.dev, Joerg Roedel <joro@8bytes.org>,
linux-pci@vger.kernel.org, Robin Murphy <robin.murphy@arm.com>,
Will Deacon <will@kernel.org>,
Lu Baolu <baolu.lu@linux.intel.com>,
galshalom@nvidia.com, Joerg Roedel <jroedel@suse.de>,
Kevin Tian <kevin.tian@intel.com>,
kvm@vger.kernel.org, maorg@nvidia.com, patches@lists.linux.dev,
tdave@nvidia.com, Tony Zhu <tony.zhu@intel.com>
Subject: Re: [PATCH 00/11] Fix incorrect iommu_groups with PCIe switches
Date: Tue, 1 Jul 2025 15:48:26 -0600 [thread overview]
Message-ID: <20250701154826.75a7aba6.alex.williamson@redhat.com> (raw)
In-Reply-To: <0-v1-74184c5043c6+195-pcie_switch_groups_jgg@nvidia.com>
Testing on some systems here...
I have an AMD system:
# lspci -tv
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-01.1-[01-03]----00.0-[02-03]----00.0-[03]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon Pro W5700]
| +-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
| +-00.2 Advanced Micro Devices, Inc. [AMD/ATI] Device 7316
| \-00.3 Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 USB
+-01.2-[04]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
+-02.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-02.1-[05]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)
+-02.2-[06-0b]----00.0-[07-0b]--+-01.0-[08]--+-00.0 MosChip Semiconductor Technology Ltd. MCS9922 PCIe Multi-I/O Controller
| | \-00.1 MosChip Semiconductor Technology Ltd. MCS9922 PCIe Multi-I/O Controller
| +-02.0-[09-0a]--+-00.0 Intel Corporation 82576 Gigabit Network Connection
| | \-00.1 Intel Corporation 82576 Gigabit Network Connection
| \-03.0-[0b]----00.0 Fresco Logic FL1100 USB 3.0 Host Controller
+-03.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-03.1-[0c]----00.0 JMicron Technology Corp. JMB58x AHCI SATA controller
+-03.2-[0d]----00.0 Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
+-04.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-08.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
+-08.1-[0e]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Raphael
| +-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [Rembrandt/Strix]
| +-00.2 Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
| +-00.3 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
| +-00.4 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
| \-00.6 Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
+-08.3-[0f]----00.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7
Notably, each case where there's a dummy host bridge followed by some
number of additional functions (ie. 01.0, 02.0, 03.0, 08.0), that dummy
host bridge is tainting the function isolation and merging the group.
For instance each of these were previously a separate group and are now
combined into one group.
# lspci -vvvs 00:01. [manually edited]
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
Capabilities: [58] Express (v2) Root Port (Slot+), IntMsgNum 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge (prog-if 00 [Normal decode])
Capabilities: [58] Express (v2) Root Port (Slot+), IntMsgNum 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
The endpoints result in equivalent grouping, but this is a case where I
don't understand how we have non-isolated functions yet isolated
subordinate buses.
An Alder Lake system shows something similar:
# lspci -tv
-[0000:00]-+-00.0 Intel Corporation 12th Gen Core Processor Host Bridge
+-01.0-[01-02]----00.0-[02]--
+-02.0 Intel Corporation Alder Lake-S GT1 [UHD Graphics 770]
+-04.0 Intel Corporation Alder Lake Innovation Platform Framework Processor Participant
+-06.0-[03]----00.0 Sandisk Corp SanDisk Ultra 3D / WD PC SN530, IX SN530, Blue SN550 NVMe SSD (DRAM-less)
+-08.0 Intel Corporation 12th Gen Core Processor Gaussian & Neural Accelerator
+-14.0 Intel Corporation Raptor Lake USB 3.2 Gen 2x2 (20 Gb/s) XHCI Host Controller
+-14.2 Intel Corporation Raptor Lake-S PCH Shared SRAM
+-15.0 Intel Corporation Raptor Lake Serial IO I2C Host Controller #0
+-15.1 Intel Corporation Raptor Lake Serial IO I2C Host Controller #1
+-15.2 Intel Corporation Raptor Lake Serial IO I2C Host Controller #2
+-15.3 Intel Corporation Device 7a4f
+-16.0 Intel Corporation Raptor Lake CSME HECI #1
+-17.0 Intel Corporation Raptor Lake SATA AHCI Controller
+-19.0 Intel Corporation Device 7a7c
+-19.1 Intel Corporation Device 7a7d
+-1a.0-[04]----00.0 Sandisk Corp SanDisk Ultra 3D / WD PC SN530, IX SN530, Blue SN550 NVMe SSD (DRAM-less)
+-1c.0-[05]--
+-1c.1-[06]----00.0 Fresco Logic FL1100 USB 3.0 Host Controller
+-1c.2-[07]----00.0 Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
+-1c.3-[08-0c]----00.0-[09-0c]--+-01.0-[0a]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
| +-02.0-[0b]--
| \-03.0-[0c]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
+-1f.0 Intel Corporation Device 7a06
+-1f.3 Intel Corporation Raptor Lake High Definition Audio Controller
+-1f.4 Intel Corporation Raptor Lake-S PCH SMBus Controller
\-1f.5 Intel Corporation Raptor Lake SPI (flash) Controller
00:1c. are all grouped together. Here 1c.0 does not report ACS, but
the other root ports do:
# lspci -vvvs 1c. | grep -e ^0 -e "Access Control Services"
00:1c.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #1 (rev 11) (prog-if 00 [Normal decode])
00:1c.1 PCI bridge: Intel Corporation Device 7a39 (rev 11) (prog-if 00 [Normal decode])
Capabilities: [220 v1] Access Control Services
00:1c.2 PCI bridge: Intel Corporation Raptor Point-S PCH - PCI Express Root Port 3 (rev 11) (prog-if 00 [Normal decode])
Capabilities: [220 v1] Access Control Services
00:1c.3 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #4 (rev 11) (prog-if 00 [Normal decode])
Capabilities: [220 v1] Access Control Services
So again the group is tainted by a device that cannot generate DMA, the
endpoint grouping remains equivalent, but isolated buses downstream of
this non-isolated group doesn't seem to make sense.
I'll try to generate further interesting configs. Thanks,
Alex
next prev parent reply other threads:[~2025-07-01 21:48 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-06-30 22:28 [PATCH 00/11] Fix incorrect iommu_groups with PCIe switches Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 01/11] PCI: Move REQ_ACS_FLAGS into pci_regs.h as PCI_ACS_ISOLATED Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 02/11] PCI: Add pci_bus_isolation() Jason Gunthorpe
2025-07-01 19:28 ` Alex Williamson
2025-07-02 1:00 ` Jason Gunthorpe
2025-07-03 15:30 ` Jason Gunthorpe
2025-07-03 22:17 ` Alex Williamson
2025-07-03 23:08 ` Alex Williamson
2025-07-03 23:21 ` Jason Gunthorpe
2025-07-03 23:15 ` Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 03/11] iommu: Compute iommu_groups properly for PCIe switches Jason Gunthorpe
2025-07-01 19:29 ` Alex Williamson
2025-07-02 1:04 ` Jason Gunthorpe
2025-07-17 19:25 ` Donald Dutile
2025-07-17 20:27 ` Jason Gunthorpe
2025-07-18 2:31 ` Donald Dutile
2025-07-18 13:32 ` Jason Gunthorpe
2025-09-22 22:32 ` Alex Williamson
2025-09-22 23:15 ` Jason Gunthorpe
2025-09-23 0:51 ` Donald Dutile
2025-09-23 1:17 ` Alex Williamson
2025-09-23 1:10 ` Alex Williamson
2025-09-23 2:26 ` Donald Dutile
2025-09-23 2:50 ` Alex Williamson
2025-09-23 12:32 ` Jason Gunthorpe
2025-09-23 12:58 ` Alex Williamson
2025-09-23 13:03 ` Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 04/11] iommu: Organize iommu_group by member size Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 05/11] PCI: Add pci_reachable_set() Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 06/11] iommu: Use pci_reachable_set() in pci_device_group() Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 07/11] iommu: Validate that pci_for_each_dma_alias() matches the groups Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 08/11] PCI: Add the ACS Enhanced Capability definitions Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 09/11] PCI: Enable ACS Enhanced bits for enable_acs and config_acs Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 10/11] PCI: Check ACS DSP/USP redirect bits in pci_enable_pasid() Jason Gunthorpe
2025-06-30 22:28 ` [PATCH 11/11] PCI: Check ACS Extended flags for pci_bus_isolated() Jason Gunthorpe
2025-07-01 21:48 ` Alex Williamson [this message]
2025-07-02 1:47 ` [PATCH 00/11] Fix incorrect iommu_groups with PCIe switches Jason Gunthorpe
2025-07-04 0:37 ` Jason Gunthorpe
2025-07-11 14:55 ` Alex Williamson
2025-07-11 16:08 ` Jason Gunthorpe
2025-07-08 20:47 ` Jason Gunthorpe
2025-07-11 15:40 ` Alex Williamson
2025-07-11 16:14 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250701154826.75a7aba6.alex.williamson@redhat.com \
--to=alex.williamson@redhat.com \
--cc=baolu.lu@linux.intel.com \
--cc=bhelgaas@google.com \
--cc=galshalom@nvidia.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=jroedel@suse.de \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=maorg@nvidia.com \
--cc=patches@lists.linux.dev \
--cc=robin.murphy@arm.com \
--cc=tdave@nvidia.com \
--cc=tony.zhu@intel.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).