From: Srirangan Madhavan <smadhavan@nvidia.com>
To: Alison Schofield <alison.schofield@intel.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Dan Williams <djbw@kernel.org>, Dave Jiang <dave.jiang@intel.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <jic23@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: Alex Williamson <alex.williamson@redhat.com>,
vsethi@nvidia.com, alwilliamson@nvidia.com,
Dan Williams <danwilliams@nvidia.com>,
Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
Vishal Aslot <vaslot@nvidia.com>,
Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
Richard Cheng <icheng@nvidia.com>,
linux-tegra@vger.kernel.org,
Srirangan Madhavan <smadhavan@nvidia.com>
Subject: [PATCH v8 00/10] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Fri, 3 Jul 2026 22:04:58 +0000 [thread overview]
Message-ID: <20260703220508.546528-1-smadhavan@nvidia.com> (raw)
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the
existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
specification [1], including CXL.cache disable, optional cache
writeback, CXL Reset initiation, ResetComplete polling, and ResetError
reporting.
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../reset
Userspace can select "cxl_reset" in reset_method and then trigger reset
through the existing reset attribute.
Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI
with the existing PCI reset_method interface.
v8 changes from v7 [2]:
- Drop the PCI helper export patch.
- Drop the multi-function sibling coordination patch. cxl_reset is only
exposed as a function-scoped reset method when the CXL reset scope is
limited to the target function.
- Keep the reset-scope discovery needed to reject non-function-scoped
CXL resets.
- Cache HDM location as BAR-relative metadata instead of keeping an
enum-time MMIO mapping.
- Restore HDM through a temporary mapping based on the current BAR
assignment after PCI config state is restored.
- Cache raw HDM decoder register state so uncommitted decoders can be
restored as uncommitted, while committed decoders are recommitted
through the normal HDM commit flow.
- Share HDM decoder decode and validation between normal CXL core
enumeration and early PCI HDM caching.
- Put cxl_reset ahead of FLR in reset_method priority because FLR does
not reset CXL.cache or CXL.mem protocol state.
Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
reset methods.
- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
the architectural reset mechanism for those protocols.
- The PCI reset_method ABI lets userspace select this narrower CXL reset
before falling back to broader bus reset methods.
Change Description:
-------------------
Patch 1: cxl/hdm: Split decoder programming into a reusable helper
- Move shared decoder settings to include/cxl/cxl.h.
- Factor low-level HDM register programming into cxl_commit().
Patch 2: cxl/hdm: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.
Patch 3: cxl/hdm: Share HDM decoder decode logic
- Share HDM decoder decode and validation between normal CXL core
enumeration and early PCI HDM cache setup.
Patch 4: cxl/hdm: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI capability initialization.
- Cache the HDM register locator as BAR-relative metadata.
- Cache raw decoder register state in addition to committed decoder
settings.
Patch 5: PCI/CXL: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate
CXL Reset, and wait for completion.
Patch 6: PCI/CXL: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges.
- Reject reset if affected ranges are busy.
- Invalidate CPU caches when runtime cache-invalidation support is
available, otherwise continue after warning.
Patch 7: PCI/CXL: Discover the CXL reset scope
- Discover whether CXL Reset is function-scoped using the Non-CXL
Function Map and CXL cache/mem capability bits.
Patch 8: cxl/pci: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset.
- Re-map HDM registers from the current BAR assignment during restore.
- Replay raw decoder state for uncommitted decoders and recommit
decoders that were committed before reset.
Patch 9: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset_method table for Type 2 reset-capable
CXL devices.
- Prioritize cxl_reset ahead of FLR.
Patch 10: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.
The CPU cache invalidation step depends on
cpu_cache_invalidate_memregion() support for the affected address ranges.
If no runtime provider is available, the kernel emits a warning and
continues after the affected HDM ranges have been reserved.
Example:
echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset
Testing:
- Ran 100 iterations of cxl_reset through the PCI reset sysfs ABI on a
CXL Type 2 device. All iterations completed successfully and
ResetComplete was observed.
- Exercised cxl_bus reset separately with an add-on HDM restore patch.
References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/
[3] https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/
Srirangan Madhavan (10):
cxl/hdm: Split decoder programming into a reusable helper
cxl/hdm: Cache decoder settings on PCI devices
cxl/hdm: Share HDM decoder decode logic
cxl/hdm: Cache endpoint decoder settings during PCI enumeration
PCI/CXL: Add CXL Device Reset helper
PCI/CXL: Validate HDM ranges before CXL reset
PCI/CXL: Discover the CXL reset scope
cxl/pci: Restore CXL HDM state after PCI reset
PCI/CXL: Expose CXL Reset as a PCI reset method
Documentation/ABI: Document CXL Reset PCI reset method
Documentation/ABI/testing/sysfs-bus-pci | 14 +
drivers/cxl/Kconfig | 4 +
drivers/cxl/core/Makefile | 2 +-
drivers/cxl/core/hdm.c | 257 ++---
drivers/cxl/core/region.c | 6 +-
drivers/cxl/core/regs.c | 4 +
drivers/cxl/core/reset.c | 1354 +++++++++++++++++++++++
drivers/cxl/cxl.h | 43 -
drivers/pci/pci.c | 2 +
drivers/pci/probe.c | 3 +
include/cxl/cxl.h | 102 +-
include/linux/pci.h | 8 +-
include/uapi/linux/pci_regs.h | 15 +
tools/testing/cxl/test/cxl.c | 10 +-
14 files changed, 1615 insertions(+), 209 deletions(-)
create mode 100644 drivers/cxl/core/reset.c
base-commit: 90cf2e0d702c8a132ccbe72e7687f33c04c14658
--
2.43.0
next reply other threads:[~2026-07-03 22:05 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-03 22:04 Srirangan Madhavan [this message]
2026-07-03 22:04 ` [PATCH v8 01/10] cxl: Split decoder programming into a reusable helper Srirangan Madhavan
2026-07-07 0:22 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 02/10] cxl: Cache decoder settings on PCI devices Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 03/10] cxl: Share HDM decoder decode logic Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 04/10] cxl: Cache endpoint decoder settings during PCI enumeration Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 05/10] cxl: Add CXL Device Reset helper Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 06/10] cxl: Validate HDM ranges before CXL reset Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 07/10] PCI/CXL: Discover the CXL reset scope Srirangan Madhavan
2026-07-07 1:45 ` Richard Cheng
2026-07-03 22:05 ` [PATCH v8 08/10] cxl: Restore CXL HDM state after PCI reset Srirangan Madhavan
2026-07-07 1:34 ` Richard Cheng
2026-07-03 22:05 ` [PATCH v8 09/10] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 10/10] Documentation/ABI: Document CXL Reset " Srirangan Madhavan
2026-07-03 22:08 ` [PATCH RFC] PCI/CXL: Restore HDM state after CXL bus reset Srirangan Madhavan
2026-07-06 23:13 ` [PATCH v8 00/10] PCI/CXL: Add CXL reset support for Type 2 devices Dave Jiang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260703220508.546528-1-smadhavan@nvidia.com \
--to=smadhavan@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=alison.schofield@intel.com \
--cc=alwilliamson@nvidia.com \
--cc=bhelgaas@google.com \
--cc=danwilliams@nvidia.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=djbw@kernel.org \
--cc=icheng@nvidia.com \
--cc=ira.weiny@intel.com \
--cc=jan@nvidia.com \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mhonap@nvidia.com \
--cc=skancherla@nvidia.com \
--cc=vaslot@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox