Linux Tegra architecture development
 help / color / mirror / Atom feed
From: Alison Schofield <alison.schofield@intel.com>
To: Srirangan Madhavan <smadhavan@nvidia.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	Dave Jiang <dave.jiang@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Ira Weiny <iweiny@kernel.org>,
	"Jonathan Cameron" <jic23@kernel.org>,
	Vishal Verma <vishal.l.verma@intel.com>,
	<linux-cxl@vger.kernel.org>, <linux-pci@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, Li Ming <ming.li@zohomail.com>,
	"Joshua Machauer" <machaujo@amazon.de>,
	Alex Williamson <alex.williamson@redhat.com>, <vsethi@nvidia.com>,
	<alwilliamson@nvidia.com>,
	Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
	Vishal Aslot <vaslot@nvidia.com>,
	Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
	Richard Cheng <icheng@nvidia.com>, <linux-tegra@vger.kernel.org>
Subject: Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Tue, 17 Nov 2026 09:28:01 -0800	[thread overview]
Message-ID: <avyOoQjiD7Uh_3rE@aschofie-mobl2.lan> (raw)
In-Reply-To: <20261105185917.2667656-1-smadhavan@nvidia.com>

On Thu, Nov 05, 2026 at 06:58:58PM +0000, Srirangan Madhavan wrote:
> Hi folks,
> 
> This series adds CXL Reset for Type 2 devices through the existing PCI
> reset_method and reset sysfs attributes. It disables CXL.cache, performs
> supported cache writeback, initiates CXL Reset with Memory Clear, and
> restores cached HDM decoder and CXL Device DVSEC state before PCI use
> resumes. Active HDM ranges are reserved and CPU caches invalidated before
> and after reset. The method remains limited to eligible single-function
> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
> the same cached state and flushes the host and device caches around the
> reset.
> 
> The reset sequence follows the CXL 4.0 specification [4], including
> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
> define the completion timeout encodings and reset access restrictions.
> 
> The userspace ABI is the existing PCI reset interface:
> 
>     /sys/bus/pci/devices/.../reset_method
>     /sys/bus/pci/devices/.../reset
> 
> Userspace selects "cxl_reset" in reset_method and triggers the reset
> through reset. Following Dan's v6 feedback, this uses the PCI interface
> instead of adding a memdev-specific sysfs ABI.
> 
> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
> callback when provided and disables PCI I/O, memory decoding, and bus
> mastering. The CXL path reserves affected HDM ranges exclusively and
> blocks IOMMU translations where applicable. It does not offline memory
> or revoke mappings: userspace is expected to offline CXL.mem users before
> reset. The reservation enforces this for claimed ranges by failing the
> reset while a conflicting resource is claimed [9].
> 
> The global region lock is held from snapshot through restoration. Memory
> Clear completion is awaited for the timeout the device advertises, up to
> 100 seconds, so the lock can be held for that long plus cache flushes.

Hi Srirangan,

Thanks for all the followups on review feedback. I think you answer some
of my question above, 'For sysfs reset...' Could you explain the intended
use cases for this reset method? Like why and when would userspace issue
a CXL reset. In particular, is reset intended to be available before endpoint
and region discovery have completed?

I see how the series saves state for restoration after reset. I don't see
what ensures discovery has progressed far enough for reset to be safe.

I also see DaveJ's cxl_bus deadlock report. My concern is related but
separate. Even without a deadlock, a reset could hold the global region lock
for up to 100 seconds plus cache maintenance and restoration, blocking
discovery and region assembly on unrelated devices. Understanding the intended
usage would help assess whether that disruption is acceptable. Note that my
simple concern is a user issuing those cxl_reset's on repeat and blocking CXL
bringup. Yeah, that's a bad user, but is there any protection against that?

Adding specific questions on patches 11, 13, and 16

-- Alison



> 
> Changes from v16 [1]:
> - Publish the PCI HDM cache under the PCI device lock instead of
>   cxl_rwsem.dpa, and drop the cxl_hdm_cache_present() and
>   cxl_hdm_cache_publish() helpers (Li Ming) [6][7]
> - Wait for the reset timeout the device advertises, up to 100 seconds,
>   instead of clamping it to 20 seconds, so Control2 is not accessed while
>   the device may still be resetting (Richard) [8]
> - State that userspace is expected to offline CXL.mem users before reset,
>   and what the HPA reservation does and does not cover (Richard) [9]
> - Enable bus mastering after the IOMMU is restored on CXL bus reset
>   (Richard) [10]
> - Explain why cxl_set_mem_enable() refreshes the cached DVSEC Control when
>   MEM_ENABLE already matches (Richard) [11]
> - Initialize the zero-size HDM decoder end to base - 1 and document the
>   zero-size case (Richard) [12]
> - Set Lock On Commit after programming the decoder registers and before
>   Commit when recommitting a locked decoder (Alison) [13]
> - Record Normalized Addressing in the cached decoder when region discovery
>   sets it, and note the short window before that update (Sashiko) [15]
> - Reject an all-ones DVSEC Control read in cxl_set_mem_enable() (Sashiko)
>   [15]
> - Add the missing context-analysis annotations on the region quiesce
>   lock and unlock and on __cxl_reset_execute() (Sashiko) [15]
> - Use an explicit pci_dev_put() in read_cdat_data() instead of mixing
>   __free() with goto cleanup (Sashiko) [15]
> - Check for VFs and multifunction devices before the CXL DVSEC lookups in
>   the two reset methods, to avoid repeated capability walks while probing
>   VFs (Joshua) [18]
> - Fold the CXL bus reset cache flush patch into this series (Dave); the
>   series is now 19 patches
> - Added Richard's Reviewed-by tags to patches 6-10, 14 and 16
> - Specified the reviewer behind each change in the changelogs, per Alison [14]
> 
> Earlier revision history is in the v16 [1] and v15 [2] cover letters.
> 
> Attribution for Dave Jiang's reference branch is still open.
> Since I am unsure about [16] I have not yet added the right tags.
> The patches link to his branch below the separator for now.
> 
> The v15 discussion on the scope check is still pending [17].
> 
> HDM caching requires CXL.cache, CXL.mem, CXL Reset and Reset Memory
> Clear capabilities. CXL bus reset state restoration uses that cache.
> 
> CXL Reset requires a registered CPU cache clean/invalidate provider
> and matching firmware, such as the separate SMCCC provider series [3].
> Without that provider, cxl_reset can be advertised while execution returns
> -ENOTTY, unless the test-only CXL_REGION_INVALIDATION_TEST bypass is
> enabled.
> 
> The KUnit suite checks HDM range boundaries, zero-size and uncommitted
> state, lock and target-type decoding, and 12-way interleave.
> 
> Testing: v17 was tested on an ARM64 target with v5 of the SMCCC
> cache provider series [3] applied. Reset loops passed for both cxl_reset and
> cxl_bus after running bidirectional host-device traffic. The
> cxl_hdm_unpack_kunit suite passes.
> 
> [1] https://lore.kernel.org/linux-cxl/20261029145703.2185231-1-smadhavan@nvidia.com/
> [2] https://lore.kernel.org/linux-cxl/20261011021422.3428136-1-smadhavan@nvidia.com/
> [3] https://lore.kernel.org/linux-arm-kernel/20261021224557.1827361-1-smadhavan@nvidia.com/
> [4] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
> [5] https://lore.kernel.org/all/cover.1791949910.git.alison.schofield@intel.com/
> [6] https://lore.kernel.org/linux-cxl/d581f2a0-dd9e-4d0a-bd19-639c1bd9b832@zohomail.com/
> [7] https://lore.kernel.org/linux-cxl/20261011021422.3428136-1-smadhavan@nvidia.com/T/#m7f8d0e51d4cfe09fefcebd93d49dcb916be9198a
> [8] https://lore.kernel.org/linux-cxl/auQC2UECQ8vaelPz@MWDK4CY14F/
> [9] https://lore.kernel.org/linux-cxl/auQRSOGWEDzB_qjF@MWDK4CY14F/
> [10] https://lore.kernel.org/linux-cxl/auRhJa-nzR0CnKf3@MWDK4CY14F/
> [11] https://lore.kernel.org/linux-cxl/auRQN6a3QVtqjd5a@MWDK4CY14F/
> [12] https://lore.kernel.org/linux-cxl/auQ27s1rSCcbdRJ3@MWDK4CY14F/
> [13] https://lore.kernel.org/linux-cxl/auAg6MFDGyVQvQxM@aschofie-mobl2.lan/
> [14] https://lore.kernel.org/linux-cxl/auOXW3BTb8CYIh4m@aschofie-mobl2.lan/
> [15] https://sashiko.dev/#/patchset/20261029145703.2185231-1-smadhavan@nvidia.com
> [16] https://lore.kernel.org/linux-cxl/20261011021422.3428136-1-smadhavan@nvidia.com/T/#m2f61467417f422316cb15b976773944dcdcb48c7
> [17] https://lore.kernel.org/linux-cxl/20261011021422.3428136-1-smadhavan@nvidia.com/T/#m4f62ed0f038f2d136158c74c1c6c338129266131
> [18] https://lore.kernel.org/linux-cxl/20261105143611.31563-1-machaujo@amazon.de/
> 
> Srirangan Madhavan (19):
>   cxl: Clarify decoder interleave limit
>   cxl: Share CXL port upstream PCI device lookup
>   cxl: Move decoder declarations to shared header
>   cxl: Embed decoder configuration in a standalone structure
>   cxl: Introduce endpoint HDM decoder settings
>   cxl: Move HDM decoder helpers to built-in code
>   cxl: Share HDM decoder register unpacking
>   cxl: Move cxl_rwsem into built-in code
>   cxl: Refresh cached PCI HDM decoder settings
>   cxl: Build register block helpers into the kernel
>   PCI/CXL: Cache endpoint HDM state during PCI enumeration
>   PCI/CXL: Add CXL Device Reset sequencing
>   cxl: Validate and synchronize HDM ranges around reset
>   PCI/CXL: Reject reset with unsafe function scope
>   PCI/CXL: Restore CXL state after PCI reset
>   PCI/CXL: Expose CXL Reset as a PCI reset method
>   PCI/CXL: Restore CXL state after CXL bus reset
>   PCI/CXL: Flush host and device caches for CXL bus reset
>   cxl: Add KUnit checks for HDM decoder unpacking
> 
>  MAINTAINERS                          |   1 +
>  drivers/cxl/.kunitconfig             |   4 +
>  drivers/cxl/Kconfig                  |  18 +-
>  drivers/cxl/Makefile                 |   1 +
>  drivers/cxl/acpi.c                   |  26 +-
>  drivers/cxl/core/Makefile            |   2 +-
>  drivers/cxl/core/atl.c               |   4 +-
>  drivers/cxl/core/core.h              |  15 +
>  drivers/cxl/core/hdm.c               | 273 +++-----
>  drivers/cxl/core/hdm_regs.c          | 437 +++++++++++++
>  drivers/cxl/core/hdm_state.c         | 402 ++++++++++++
>  drivers/cxl/core/pci.c               |  85 ++-
>  drivers/cxl/core/port.c              |  68 +-
>  drivers/cxl/core/region.c            | 116 ++--
>  drivers/cxl/core/regs.c              |   9 +
>  drivers/cxl/cxl.h                    |  46 +-
>  drivers/cxl/pci.c                    |   2 +-
>  drivers/cxl/port.c                   |   4 +-
>  drivers/cxl/tests/Makefile           |   5 +
>  drivers/cxl/tests/hdm_unpack_kunit.c |  91 +++
>  drivers/pci/Makefile                 |   1 +
>  drivers/pci/bus.c                    |   1 +
>  drivers/pci/cxl.c                    | 907 +++++++++++++++++++++++++++
>  drivers/pci/pci.c                    |  27 +-
>  drivers/pci/pci.h                    |  44 ++
>  drivers/pci/probe.c                  |   1 +
>  include/cxl/cxl.h                    |  59 +-
>  include/cxl/hdm.h                    | 109 ++++
>  include/cxl/pci.h                    |  13 +
>  include/linux/pci.h                  |   8 +-
>  include/uapi/linux/pci_regs.h        |  14 +
>  tools/testing/cxl/Kbuild             |   1 -
>  tools/testing/cxl/test/cxl.c         | 158 +++--
>  33 files changed, 2526 insertions(+), 426 deletions(-)
>  create mode 100644 drivers/cxl/.kunitconfig
>  create mode 100644 drivers/cxl/core/hdm_regs.c
>  create mode 100644 drivers/cxl/core/hdm_state.c
>  create mode 100644 drivers/cxl/tests/Makefile
>  create mode 100644 drivers/cxl/tests/hdm_unpack_kunit.c
>  create mode 100644 drivers/pci/cxl.c
>  create mode 100644 include/cxl/hdm.h
> 
> 
> base-commit: fef22d37d47043b63e4007f2f5fe762fa21fae0c
> -- 
> 2.43.0
> 

      parent reply	other threads:[~2026-11-17 17:28 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-05 18:58 [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Srirangan Madhavan
2026-11-05 18:58 ` [PATCH v17 01/19] cxl: Clarify decoder interleave limit Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 02/19] cxl: Share CXL port upstream PCI device lookup Srirangan Madhavan
2026-11-10 13:36   ` Li Ming
2026-11-13 15:50     ` Dave Jiang
2026-11-15  3:25       ` Li Ming
2026-11-05 18:59 ` [PATCH v17 03/19] cxl: Move decoder declarations to shared header Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 04/19] cxl: Embed decoder configuration in a standalone structure Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 05/19] cxl: Introduce endpoint HDM decoder settings Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 06/19] cxl: Move HDM decoder helpers to built-in code Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 07/19] cxl: Share HDM decoder register unpacking Srirangan Madhavan
2026-11-12 21:35   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 08/19] cxl: Move cxl_rwsem into built-in code Srirangan Madhavan
2026-11-12 23:49   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 09/19] cxl: Refresh cached PCI HDM decoder settings Srirangan Madhavan
2026-11-13  0:38   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 10/19] cxl: Build register block helpers into the kernel Srirangan Madhavan
2026-11-13 18:02   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 11/19] PCI/CXL: Cache endpoint HDM state during PCI enumeration Srirangan Madhavan
2026-11-10 14:12   ` Li Ming
2026-11-11  3:54     ` Li Ming
2026-11-17 17:29   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing Srirangan Madhavan
2026-11-10 14:29   ` Li Ming
2026-11-11 10:12   ` Lucero Palau, Alejandro
2026-11-05 18:59 ` [PATCH v17 13/19] cxl: Validate and synchronize HDM ranges around reset Srirangan Madhavan
2026-11-17 17:32   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 14/19] PCI/CXL: Reject reset with unsafe function scope Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 15/19] PCI/CXL: Restore CXL state after PCI reset Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 16/19] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-11-17 17:37   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 17/19] PCI/CXL: Restore CXL state after CXL bus reset Srirangan Madhavan
2026-11-13 17:37   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 18/19] PCI/CXL: Flush host and device caches for " Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 19/19] cxl: Add KUnit checks for HDM decoder unpacking Srirangan Madhavan
2026-11-05 19:31 ` [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Alison Schofield
2026-11-16 19:11 ` Dave Jiang
2026-11-17 17:28 ` Alison Schofield [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avyOoQjiD7Uh_3rE@aschofie-mobl2.lan \
    --to=alison.schofield@intel.com \
    --cc=alex.williamson@redhat.com \
    --cc=alwilliamson@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=icheng@nvidia.com \
    --cc=iweiny@kernel.org \
    --cc=jan@nvidia.com \
    --cc=jic23@kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-tegra@vger.kernel.org \
    --cc=machaujo@amazon.de \
    --cc=mhonap@nvidia.com \
    --cc=ming.li@zohomail.com \
    --cc=skancherla@nvidia.com \
    --cc=smadhavan@nvidia.com \
    --cc=vaslot@nvidia.com \
    --cc=vishal.l.verma@intel.com \
    --cc=vsethi@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox