From: Srirangan Madhavan <smadhavan@nvidia.com>
To: Alison Schofield <alison.schofield@intel.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Dave Jiang <dave.jiang@intel.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <jic23@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: Alex Williamson <alex.williamson@redhat.com>,
vsethi@nvidia.com, alwilliamson@nvidia.com,
Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
Vishal Aslot <vaslot@nvidia.com>,
Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
Richard Cheng <icheng@nvidia.com>,
linux-tegra@vger.kernel.org,
Srirangan Madhavan <smadhavan@nvidia.com>
Subject: [PATCH v10 00/12] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Tue, 4 Aug 2026 19:29:46 +0000 [thread overview]
Message-ID: <20260804192958.1823952-1-smadhavan@nvidia.com> (raw)
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the
existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
specification [1], including CXL.cache disable, optional cache
writeback, CXL Reset initiation, ResetComplete polling, and ResetError
reporting.
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../reset
Userspace can select "cxl_reset" in reset_method and then trigger reset
through the existing reset attribute.
Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI
with the existing PCI reset_method interface.
v10 changes from v9 [2]:
- Address review comments deferred from v9.
- Split HDM decoder helper refactoring into smaller reviewable patches.
- Rename the built-in CXL helper file to resource.c and keep reset-specific
logic separate from the decoder programming refactor.
- Add a helper to resolve a CXL port's upstream PCI device.
- Fix tools/testing/cxl Kbuild coverage for resource.o.
- Keep early HDM decoder parsing shared with normal CXL HDM enumeration.
- Gate CXL Reset execution on cached HDM state and current CXL.cache/CXL.mem
enablement.
- Reject CXL Reset on multifunction devices to avoid reset scope ambiguity.
- Order CXL cache disable/writeback before PCI reset quiesce and IOMMU reset
preparation.
- Restore cached HDM state without replaying transient commit/error/lock
control bits.
- Update ABI documentation to match the implemented reset ordering and CPU
cache invalidation behavior.
v9 changes from v8 [3]:
- Rebase the series onto v7.2-rc2 so it applies cleanly on a Linus rc
tag.
- Include the cxl_bus HDM restore add-on as patch 11/11 so b4 orders it
after the main CXL reset series.
- Address only patch application issues in v9 to gather feedback on the
current direction. Other review comments, including Dave's code
structure/readability feedback, are deferred to v10.
v8 changes from v7 [4]:
- Drop the PCI helper export patch.
- Drop the multi-function sibling coordination patch. cxl_reset is only
exposed as a function-scoped reset method when the CXL reset scope is
limited to the target function.
- Keep the reset-scope discovery needed to reject non-function-scoped
CXL resets.
- Cache HDM location as BAR-relative metadata instead of keeping an
enum-time MMIO mapping.
- Restore HDM through a temporary mapping based on the current BAR
assignment after PCI config state is restored.
- Cache raw HDM decoder register state so uncommitted decoders can be
restored as uncommitted, while committed decoders are recommitted
through the normal HDM commit flow.
- Share HDM decoder decode and validation between normal CXL core
enumeration and early PCI HDM caching.
- Put cxl_reset ahead of FLR in reset_method priority because FLR does
not reset CXL.cache or CXL.mem protocol state.
Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
reset methods.
- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
the architectural reset mechanism for those protocols.
- The PCI reset_method ABI lets userspace select this narrower CXL reset
before falling back to broader bus reset methods.
Change Description:
-------------------
Patch 1: cxl: Move HDM decoder programming helpers
- Move low-level HDM decoder programming helpers into resource.c.
- Keep the movement mechanical so later patches can change the interface
separately.
Patch 2: cxl: Pass decoder settings to HDM commit helpers
- Introduce a decoder settings snapshot for HDM commit programming.
- Route decoder commit through helpers that can be reused by reset restore.
Patch 3: cxl: Share HDM decoder decode logic
- Add a small helper to resolve a CXL port's upstream PCI device.
- Share HDM decoder decode and validation between normal CXL core
enumeration and early PCI HDM cache setup.
Patch 4: cxl: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.
Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI enumeration.
- Cache the HDM register locator as BAR-relative metadata.
- Cache raw decoder register state in addition to committed decoder
settings.
Patch 6: cxl: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate
CXL Reset, and wait for completion.
Patch 7: cxl: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges.
- Reject reset if affected ranges are busy.
- Invalidate CPU caches when runtime cache-invalidation support is
available, otherwise continue after warning.
Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices
- Reject CXL Reset for multifunction devices to avoid reset scope ambiguity.
Patch 9: cxl: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset.
- Re-map HDM registers from the current BAR assignment during restore.
- Replay raw decoder state for uncommitted decoders and recommit
decoders that were committed before reset.
Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset-method table for Type 2 reset-capable
CXL devices.
- Prioritize cxl_reset ahead of FLR.
Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.
Patch 12: PCI/CXL: Restore HDM state after CXL bus reset
- Restore cached CXL HDM decoder state after cxl_bus reset.
The CPU cache invalidation step depends on
cpu_cache_invalidate_memregion() support for the affected address ranges.
If no runtime provider is available, the kernel emits a warning and
continues after the affected HDM ranges have been reserved.
Example:
echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset
Testing:
- Ran 100 iterations of cxl_reset through the PCI reset sysfs ABI on a
CXL Type 2 device. All iterations completed successfully and
ResetComplete was observed.
- Exercised cxl_bus reset separately with the HDM restore add-on patch.
- Built drivers/cxl/core/resource.o and drivers/pci/pci.o during v10
preparation.
References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/
[3] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/
[4] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/
Srirangan Madhavan (12):
cxl: Move HDM decoder programming helpers
cxl: Pass decoder settings to HDM commit helpers
cxl: Share HDM decoder decode logic
cxl: Cache decoder settings on PCI devices
cxl: Cache endpoint decoder settings during PCI enumeration
cxl: Add CXL Device Reset helper
cxl: Validate HDM ranges before CXL reset
PCI/CXL: Reject CXL Reset on multifunction devices
cxl: Restore CXL HDM state after PCI reset
PCI/CXL: Expose CXL Reset as a PCI reset method
Documentation/ABI: Document CXL Reset PCI reset method
PCI/CXL: Restore HDM state after CXL bus reset
Documentation/ABI/testing/sysfs-bus-pci | 16 +
drivers/cxl/Kconfig | 4 +
drivers/cxl/core/Makefile | 2 +-
drivers/cxl/core/core.h | 7 +
drivers/cxl/core/hdm.c | 351 +++---
drivers/cxl/core/port.c | 19 +
drivers/cxl/core/regs.c | 4 +
drivers/cxl/core/resource.c | 1341 +++++++++++++++++++++++
drivers/cxl/cxl.h | 41 -
drivers/pci/bus.c | 2 +
drivers/pci/pci.c | 8 +
drivers/pci/probe.c | 2 +
include/cxl/cxl.h | 116 +-
include/linux/pci.h | 8 +-
include/uapi/linux/pci_regs.h | 14 +
tools/testing/cxl/Kbuild | 2 +-
16 files changed, 1727 insertions(+), 210 deletions(-)
create mode 100644 drivers/cxl/core/resource.c
base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
--
2.43.0
next reply other threads:[~2026-08-04 19:30 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-04 19:29 Srirangan Madhavan [this message]
2026-08-04 19:29 ` [PATCH v10 01/12] cxl: Move HDM decoder programming helpers Srirangan Madhavan
2026-08-05 2:13 ` Alison Schofield
2026-08-04 19:29 ` [PATCH v10 02/12] cxl: Pass decoder settings to HDM commit helpers Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 03/12] cxl: Share HDM decoder decode logic Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 04/12] cxl: Cache decoder settings on PCI devices Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 05/12] cxl: Cache endpoint decoder settings during PCI enumeration Srirangan Madhavan
2026-08-05 2:28 ` Alison Schofield
2026-08-04 19:29 ` [PATCH v10 06/12] cxl: Add CXL Device Reset helper Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 07/12] cxl: Validate HDM ranges before CXL reset Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 08/12] cxl: Reject CXL Reset on multifunction devices Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 09/12] cxl: Restore CXL HDM state after PCI reset Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 10/12] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 11/12] Documentation/ABI: Document CXL Reset " Srirangan Madhavan
2026-08-04 19:29 ` [PATCH v10 12/12] PCI/CXL: Restore HDM state after CXL bus reset Srirangan Madhavan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260804192958.1823952-1-smadhavan@nvidia.com \
--to=smadhavan@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=alison.schofield@intel.com \
--cc=alwilliamson@nvidia.com \
--cc=bhelgaas@google.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=icheng@nvidia.com \
--cc=ira.weiny@intel.com \
--cc=jan@nvidia.com \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mhonap@nvidia.com \
--cc=skancherla@nvidia.com \
--cc=vaslot@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox