Re: [PATCH v11 00/12] PCI/CXL: Add CXL reset support for Type 2 devices

[Richard Cheng <icheng@nvidia.com>]

On Wed, Sep 02, 2026 at 07:27:52AM +0800, Srirangan Madhavan wrote:
> Hi folks!
> 
> This series adds CXL Reset support for CXL Type 2 devices through the
> existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
> specification [1], including CXL.cache disable, optional cache
> writeback, CXL Reset initiation, ResetComplete polling, and ResetError
> reporting.
>

Hi Srirangan,
 
> The userspace ABI is the existing PCI reset interface:
> 
>     /sys/bus/pci/devices/.../reset_method
>     /sys/bus/pci/devices/.../reset
> 
> Userspace can select "cxl_reset" in reset_method and then trigger reset
> through the existing reset attribute.
> 
> CXL Reset is exposed conservatively. The target must be a
> single-function CXL Type 2 device with CXL.cache and CXL.mem enabled,
> and the kernel must have cached HDM state sufficient to validate affected
> ranges and restore decoder programming. The reset preserves device
> memory; it does not request CXL Reset Memory Clear.
> 

It seems reasonable to me, the whole approach.
I wonder if we can document the complete reset-safety contract down ?

Especially what guarantees that all CXL.mem users and functions are quiesced ?
Besides HDM EP , what are other EP state that should be retained or restored ?

I think it will make people understand the mechanism more easily.

Best regards,
Richard Cheng.

> Following Dan's v6 feedback, this uses the existing PCI reset_method
> interface instead of adding a memdev-specific sysfs ABI.
> 
> v11 changes from v10 [2]:
> - Rebase onto v7.3-rc1, which includes the CXL Type 2 enabling work, and
>   use the merged public CXL PCI header.
> - Rename the built-in support symbol to CONFIG_CXL_RESET and fix
>   built-in/module linkage for CONFIG_CXL_BUS=m.
> - Fix tools/testing/cxl linkage without compiling production helper
>   objects twice in the mock cxl_core module.
> - Simplify decoder snapshot handling and preserve endpoint DPA skip
>   ownership under the existing DPA lock.
> - Keep policy out of shared decoder parsing by accepting spec-legal
>   zero-size decoder state, while skipping zero-size ranges during reset
>   validation.
> - Tighten early HDM cache setup, capability checks, BAR bounds, MSE
>   enable/restore handling, and cache publication locking.
> - Keep the conservative multifunction-device rejection requested in
>   earlier review.
> - Harden post-reset failure handling so MSE and BME remain disabled when
>   HDM state cannot be safely restored.
> - Serialize CXL bus-reset HDM restore against region updates and flush
>   posted HDM writes before PCI_COMMAND can restore bus mastering.
> - Address the remaining reviewability and naming feedback from v10.
> 
> Motivation:
> -----------
> - Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
>   reset methods.
> 
> - FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
>   the architectural reset mechanism for those protocols.
> 
> - The PCI reset_method ABI lets userspace select CXL Reset before
>   falling back to broader bus reset methods.
> 
> Change Description:
> -------------------
> 
> Patch 1: cxl: Move HDM decoder programming helpers
> - Introduce a decoder settings snapshot collected under the DPA lock.
> - Move low-level settings-based HDM programming into resource.c.
> 
> Patch 2: cxl: Make HDM commit helpers available to reset code
> - Build generic HDM resource helpers under CONFIG_CXL_RESET.
> - Export the lock and commit helpers needed when cxl_core is modular.
> 
> Patch 3: cxl: Share HDM decoder decode logic
> - Share HDM decoder decode and validation between normal CXL core
>   enumeration and early PCI HDM cache setup.
> - Decode zero-size committed decoder state without imposing caller policy.
> 
> Patch 4: cxl: Cache decoder settings on PCI devices
> - Cache CXL core HDM decoder settings in pci_dev->hdm.
> - Refresh the cache as decoders are enumerated, committed, or reset.
> 
> Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration
> - Snapshot endpoint HDM state during PCI enumeration.
> - Cache the HDM register locator as BAR-relative metadata and preserve raw
>   decoder state for reset restore.
> 
> Patch 6: cxl: Add CXL Device Reset helper
> - Add the internal DVSEC reset sequence.
> - Disable CXL.cache, perform cache writeback where supported, initiate
>   CXL Reset, and wait for completion.
> 
> Patch 7: cxl: Validate HDM ranges before CXL reset
> - Collect enabled cached HDM ranges, reject busy ranges, and perform CPU
>   cache invalidation when a runtime backend is available.
> 
> Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices
> - Conservatively reject CXL Reset where function scope cannot be assumed.
> 
> Patch 9: cxl: Restore CXL HDM state after PCI reset
> - Restore cached global and per-decoder HDM state after reset while the
>   target IOMMU reset block remains active.
> 
> Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
> - Add "cxl_reset" to the PCI reset-method table ahead of FLR.
> 
> Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
> - Document the new reset_method value and reset behavior.
> 
> Patch 12: PCI/CXL: Restore HDM state after CXL bus reset
> - Restore cached CXL HDM decoder state after cxl_bus reset.
> 
> The CPU cache invalidation step depends on
> cpu_cache_invalidate_memregion() support for the affected address ranges.
> If no runtime provider is available, the kernel emits a warning and
> continues after the affected HDM ranges have been reserved.
> 
> Example:
> 
>     echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
>     echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset
> 
> Testing:
> - Built every intermediate patch, plus the v7.3-rc1 baseline, with
>   CONFIG_CXL_BUS=m. All 12 patches passed a full kernel build and
>   `make M=tools/testing/cxl/` independently.
> - Built and booted the series on an arm64 NVIDIA NVL72 system and
>   successfully exercised CXL Reset through the PCI reset sysfs ABI.
> - Earlier revision testing ran 100 iterations of cxl_reset and exercised
>   cxl_bus separately with HDM state restoration.
> 
> References:
> [1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
> [2] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/
> [3] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/
> [4] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/
> [5] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/
> 
> Srirangan Madhavan (12):
>   cxl: Move HDM decoder programming helpers
>   cxl: Make HDM commit helpers available to reset code
>   cxl: Share HDM decoder decode logic
>   cxl: Cache decoder settings on PCI devices
>   cxl: Cache endpoint decoder settings during PCI enumeration
>   cxl: Add CXL Device Reset helper
>   cxl: Validate HDM ranges before CXL reset
>   PCI/CXL: Reject CXL Reset on multifunction devices
>   cxl: Restore CXL HDM state after PCI reset
>   PCI/CXL: Expose CXL Reset as a PCI reset method
>   Documentation/ABI: Document CXL Reset PCI reset method
>   PCI/CXL: Restore HDM state after CXL bus reset
> 
>  Documentation/ABI/testing/sysfs-bus-pci |   15 +
>  drivers/cxl/Kconfig                     |    4 +
>  drivers/cxl/core/Makefile               |    2 +-
>  drivers/cxl/core/core.h                 |    8 +
>  drivers/cxl/core/hdm.c                  |  324 +++---
>  drivers/cxl/core/pci.c                  |    7 +-
>  drivers/cxl/core/port.c                 |   19 +
>  drivers/cxl/core/regs.c                 |    4 +
>  drivers/cxl/core/resource.c             | 1351 +++++++++++++++++++++++
>  drivers/cxl/cxl.h                       |   41 -
>  drivers/pci/bus.c                       |    2 +
>  drivers/pci/pci.c                       |    8 +
>  drivers/pci/probe.c                     |    2 +
>  include/cxl/cxl.h                       |  116 +-
>  include/linux/pci.h                     |    8 +-
>  include/uapi/linux/pci_regs.h           |   14 +
>  tools/testing/cxl/Kbuild                |    1 -
>  17 files changed, 1715 insertions(+), 211 deletions(-)
>  create mode 100644 drivers/cxl/core/resource.c
> 
> 
> base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
> -- 
> 2.43.0