Re: [PATCH v8 00/10] PCI/CXL: Add CXL reset support for Type 2 devices

[Richard Cheng <icheng@nvidia.com>]

On Fri, Jul 03, 2026 at 10:04:58PM +0800, Srirangan Madhavan wrote:
> Hi folks!
> 
> This series adds CXL Reset support for CXL Type 2 devices through the
> existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
> specification [1], including CXL.cache disable, optional cache
> writeback, CXL Reset initiation, ResetComplete polling, and ResetError
> reporting.
> 
> The userspace ABI is the existing PCI reset interface:
> 
>     /sys/bus/pci/devices/.../reset_method
>     /sys/bus/pci/devices/.../reset
> 
> Userspace can select "cxl_reset" in reset_method and then trigger reset
> through the existing reset attribute.
> 
> Following Dan's v6 feedback, this replaces the proposed memdev sysfs ABI
> with the existing PCI reset_method interface.
> 
> v8 changes from v7 [2]:
> - Drop the PCI helper export patch.
> - Drop the multi-function sibling coordination patch. cxl_reset is only
>   exposed as a function-scoped reset method when the CXL reset scope is
>   limited to the target function.
> - Keep the reset-scope discovery needed to reject non-function-scoped
>   CXL resets.
> - Cache HDM location as BAR-relative metadata instead of keeping an
>   enum-time MMIO mapping.
> - Restore HDM through a temporary mapping based on the current BAR
>   assignment after PCI config state is restored.
> - Cache raw HDM decoder register state so uncommitted decoders can be
>   restored as uncommitted, while committed decoders are recommitted
>   through the normal HDM commit flow.
> - Share HDM decoder decode and validation between normal CXL core
>   enumeration and early PCI HDM caching.
> - Put cxl_reset ahead of FLR in reset_method priority because FLR does
>   not reset CXL.cache or CXL.mem protocol state.
>

Hi Srirangan,

Thanks for this, I have some questions and suggestions in your patches,
but overall I let codex ran 260 cxl_reset on a machine with 2 CXL-capable
GPUs, exposed as Type 2 device, and 0 failures for this series, at least
on my HW it works.

Let me summarize what I did
- 100 sequential resets on each GPU, 200 in totoal, all rc=0, device healthy after each.
- 20 rounds of concurrent reset on both GPUs, plus 40 iterations alternating reset_method: 0 failures.
- No WARN/BUG/soft-lockup/refcount, no leaked in /proc/iomem afterward, DVSEC CTRL2/STATUS2 stable across pre/post

I might run more test with KASAN when I have time, but so far it looks good functionally.

Thanks for your work.

Best regards,
Richard Cheng

 
> Motivation:
> -----------
> - Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
>   reset methods.
> 
> - FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
>   the architectural reset mechanism for those protocols.
> 
> - The PCI reset_method ABI lets userspace select this narrower CXL reset
>   before falling back to broader bus reset methods.
> 
> Change Description:
> -------------------
> 
> Patch 1: cxl/hdm: Split decoder programming into a reusable helper
> - Move shared decoder settings to include/cxl/cxl.h.
> - Factor low-level HDM register programming into cxl_commit().
> 
> Patch 2: cxl/hdm: Cache decoder settings on PCI devices
> - Cache CXL core HDM decoder settings in pci_dev->hdm.
> - Refresh the cache as decoders are enumerated, committed, or reset.
> 
> Patch 3: cxl/hdm: Share HDM decoder decode logic
> - Share HDM decoder decode and validation between normal CXL core
>   enumeration and early PCI HDM cache setup.
> 
> Patch 4: cxl/hdm: Cache endpoint decoder settings during PCI enumeration
> - Snapshot endpoint HDM state during PCI capability initialization.
> - Cache the HDM register locator as BAR-relative metadata.
> - Cache raw decoder register state in addition to committed decoder
>   settings.
> 
> Patch 5: PCI/CXL: Add CXL Device Reset helper
> - Add the internal DVSEC reset sequence.
> - Disable CXL.cache, perform cache writeback where supported, initiate
>   CXL Reset, and wait for completion.
> 
> Patch 6: PCI/CXL: Validate HDM ranges before CXL reset
> - Collect enabled cached HDM ranges.
> - Reject reset if affected ranges are busy.
> - Invalidate CPU caches when runtime cache-invalidation support is
>   available, otherwise continue after warning.
> 
> Patch 7: PCI/CXL: Discover the CXL reset scope
> - Discover whether CXL Reset is function-scoped using the Non-CXL
>   Function Map and CXL cache/mem capability bits.
> 
> Patch 8: cxl/pci: Restore CXL HDM state after PCI reset
> - Restore cached global and per-decoder HDM state after reset.
> - Re-map HDM registers from the current BAR assignment during restore.
> - Replay raw decoder state for uncommitted decoders and recommit
>   decoders that were committed before reset.
> 
> Patch 9: PCI/CXL: Expose CXL Reset as a PCI reset method
> - Add "cxl_reset" to the PCI reset_method table for Type 2 reset-capable
>   CXL devices.
> - Prioritize cxl_reset ahead of FLR.
> 
> Patch 10: Documentation/ABI: Document CXL Reset PCI reset method
> - Document the new reset_method value and reset behavior.
> 
> The CPU cache invalidation step depends on
> cpu_cache_invalidate_memregion() support for the affected address ranges.
> If no runtime provider is available, the kernel emits a warning and
> continues after the affected HDM ranges have been reserved.
> 
> Example:
> 
>     echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
>     echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset
> 
> Testing:
> - Ran 100 iterations of cxl_reset through the PCI reset sysfs ABI on a
>   CXL Type 2 device. All iterations completed successfully and
>   ResetComplete was observed.
> - Exercised cxl_bus reset separately with an add-on HDM restore patch.
> 
> References:
> [1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
> [2] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/
> [3] https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/
> 
> Srirangan Madhavan (10):
>   cxl/hdm: Split decoder programming into a reusable helper
>   cxl/hdm: Cache decoder settings on PCI devices
>   cxl/hdm: Share HDM decoder decode logic
>   cxl/hdm: Cache endpoint decoder settings during PCI enumeration
>   PCI/CXL: Add CXL Device Reset helper
>   PCI/CXL: Validate HDM ranges before CXL reset
>   PCI/CXL: Discover the CXL reset scope
>   cxl/pci: Restore CXL HDM state after PCI reset
>   PCI/CXL: Expose CXL Reset as a PCI reset method
>   Documentation/ABI: Document CXL Reset PCI reset method
> 
>  Documentation/ABI/testing/sysfs-bus-pci |   14 +
>  drivers/cxl/Kconfig                     |    4 +
>  drivers/cxl/core/Makefile               |    2 +-
>  drivers/cxl/core/hdm.c                  |  257 ++---
>  drivers/cxl/core/region.c               |    6 +-
>  drivers/cxl/core/regs.c                 |    4 +
>  drivers/cxl/core/reset.c                | 1354 +++++++++++++++++++++++
>  drivers/cxl/cxl.h                       |   43 -
>  drivers/pci/pci.c                       |    2 +
>  drivers/pci/probe.c                     |    3 +
>  include/cxl/cxl.h                       |  102 +-
>  include/linux/pci.h                     |    8 +-
>  include/uapi/linux/pci_regs.h           |   15 +
>  tools/testing/cxl/test/cxl.c            |   10 +-
>  14 files changed, 1615 insertions(+), 209 deletions(-)
>  create mode 100644 drivers/cxl/core/reset.c
> 
> base-commit: 90cf2e0d702c8a132ccbe72e7687f33c04c14658
> -- 
> 2.43.0