From: Dave Jiang <dave.jiang@intel.com>
To: Srirangan Madhavan <smadhavan@nvidia.com>,
Alison Schofield <alison.schofield@intel.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <iweiny@kernel.org>,
Jonathan Cameron <jic23@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: Li Ming <ming.li@zohomail.com>,
Joshua Machauer <machaujo@amazon.de>,
Alex Williamson <alex.williamson@redhat.com>,
vsethi@nvidia.com, alwilliamson@nvidia.com,
Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
Vishal Aslot <vaslot@nvidia.com>,
Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
Richard Cheng <icheng@nvidia.com>,
linux-tegra@vger.kernel.org
Subject: Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Mon, 16 Nov 2026 12:11:00 -0700 [thread overview]
Message-ID: <5616c825-454e-4eba-9352-b06d98dc123a@intel.com> (raw)
In-Reply-To: <20261105185917.2667656-1-smadhavan@nvidia.com>
On 11/5/26 11:58 AM, Srirangan Madhavan wrote:
> Hi folks,
>
> This series adds CXL Reset for Type 2 devices through the existing PCI
> reset_method and reset sysfs attributes. It disables CXL.cache, performs
> supported cache writeback, initiates CXL Reset with Memory Clear, and
> restores cached HDM decoder and CXL Device DVSEC state before PCI use
> resumes. Active HDM ranges are reserved and CPU caches invalidated before
> and after reset. The method remains limited to eligible single-function
> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
> the same cached state and flushes the host and device caches around the
> reset.
>
> The reset sequence follows the CXL 4.0 specification [4], including
> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
> define the completion timeout encodings and reset access restrictions.
>
> The userspace ABI is the existing PCI reset interface:
>
> /sys/bus/pci/devices/.../reset_method
> /sys/bus/pci/devices/.../reset
>
> Userspace selects "cxl_reset" in reset_method and triggers the reset
> through reset. Following Dan's v6 feedback, this uses the PCI interface
> instead of adding a memdev-specific sysfs ABI.
>
> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
> callback when provided and disables PCI I/O, memory decoding, and bus
> mastering. The CXL path reserves affected HDM ranges exclusively and
> blocks IOMMU translations where applicable. It does not offline memory
> or revoke mappings: userspace is expected to offline CXL.mem users before
> reset. The reservation enforces this for claimed ranges by failing the
> reset while a conflicting resource is claimed [9].
>
> The global region lock is held from snapshot through restoration. Memory
> Clear completion is awaited for the timeout the device advertises, up to
> 100 seconds, so the lock can be held for that long plus cache flushes.
I think holding it across the bus reset can deadlock with pciehp and a
PCI remove. I tested in QEMU with two cxl-type2 devices from Manish's
RFC [1].
For a device in a pciehp slot, pci_reset_bus_function() ends up in
pciehp_reset_slot(), which takes the slot's reset_lock. The reset still
holds cxl_rwsem.region at that point.
Two other tasks can complete a cycle. pciehp_ist() for that slot can
hold reset_lock and wait for pci_rescan_remove_lock in
pciehp_unconfigure_device(). A PCI remove of a second Type 2 device can
hold pci_rescan_remove_lock while cxl_decoder_detach() waits for
cxl_rwsem.region.
On v17 as posted, lockdep reports the cycle:
[ 60.891789] WARNING: possible circular locking dependency detected
[ 60.893014] irq/24-pciehp/103 is trying to acquire lock:
[ 60.893367] ffff888072b4f128 (&cxl_memdev_key){+.+.}-{4:4}, at: device_del+0x8d/0x580
but task is already holding lock:
[ 60.894498] ffffffff84699e80 (pci_rescan_remove_lock){+.+.}-{4:4}, at: pciehp_unconfigure_device+0x93/0x210
[...]
-> #3 (&ctrl->reset_lock){++++}-{4:4}:
[ 60.916794] down_write_nested+0x80/0x170
[ 60.917826] pciehp_reset_slot+0x6c/0xc0
[ 60.918893] pci_reset_hotplug_slot+0x5c/0x90
[ 60.919928] pci_reset_bus_function+0x161/0x330
[ 60.920951] cxl_reset_bus_function+0x1c6/0x280
[ 60.921958] __pci_reset_function_locked+0x96/0x170
[ 60.922990] pci_reset_function+0xc8/0x180
[ 60.923977] reset_store+0x84/0xd0
[...]
-> #2 (cxl_rwsem.region){++++}-{4:4}:
[...]
[ 61.013490] Chain exists of:
&cxl_memdev_key --> &ctrl->reset_lock --> pci_rescan_remove_lock
With a test-only msleep() after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(), it turns into a real hang. The reset, a sysfs
remove of the second device and a power off of the first device's slot
all got stuck.
Two region reads blocked behind the reset as well. A read of
region1/commit timed out after 5 s, and cxl list -R stayed in D state in
decoders_committed_show().
To reproduce, I used Manish's RFC as posted. It allows one cxl-type2 per
pxb-cxl, so each device has its own host bridge and window. On q35 with
cxl=on, plus the memory backends:
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=53
-device cxl-rp,id=hb0rp0,bus=cxl.0,chassis=0,slot=0,port=0
-device cxl-type2,bus=hb0rp0,volatile-memdev=t2mem0,id=cxl-t2,fmw-index=0
-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191
-device cxl-rp,id=hb1rp0,bus=cxl.1,chassis=0,slot=2,port=0
-device cxl-type2,bus=hb1rp0,volatile-memdev=t2mem1,id=cxl-t2b,fmw-index=1
-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G
The guest kernel needs CONFIG_CXL_REGION_INVALIDATION_TEST=y. Without it,
the cxl_bus reset fails in an x86 guest with "failed to prepare CXL bus
reset ranges: -6".
I drove both devices with a small test module based on
tools/testing/cxl/test/accel.c. It attaches each firmware-committed
region with devm_cxl_probe_mem() and has no reset handlers. Then, with
lockdep on:
echo cxl_bus > /sys/bus/pci/devices/0000:36:00.0/reset_method
echo 1 > /sys/bus/pci/devices/0000:36:00.0/reset
echo 0 > /sys/bus/pci/slots/2/power # second device, 0000:c0:00.0
For the hang, add msleep(10000) after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(). Start the cxl_bus reset, remove 0000:c0:00.0
through sysfs two seconds later, and power off slot 0 two seconds after
that.
Separately, pci_cxl_reset_prepare() returns with the quiesce held on its
success path only, and cxl_restore_state_after_pci_reset() releases it.
With Clang 23 and CONTEXT_ANALYSIS=y passed to make, that gives five
cxl_region_quiesce warnings in drivers/pci/cxl.c.
[1]: https://lore.kernel.org/all/20261106035911.1280066-1-mhonap@nvidia.com/
DJ
next prev parent reply other threads:[~2026-11-16 19:11 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 18:58 [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Srirangan Madhavan
2026-11-05 18:58 ` [PATCH v17 01/19] cxl: Clarify decoder interleave limit Srirangan Madhavan
2026-11-05 19:03 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 02/19] cxl: Share CXL port upstream PCI device lookup Srirangan Madhavan
2026-11-05 19:12 ` sashiko-bot
2026-11-10 13:36 ` Li Ming
2026-11-13 15:50 ` Dave Jiang
2026-11-15 3:25 ` Li Ming
2026-11-05 18:59 ` [PATCH v17 03/19] cxl: Move decoder declarations to shared header Srirangan Madhavan
2026-11-05 19:06 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 04/19] cxl: Embed decoder configuration in a standalone structure Srirangan Madhavan
2026-11-05 19:07 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 05/19] cxl: Introduce endpoint HDM decoder settings Srirangan Madhavan
2026-11-05 19:09 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 06/19] cxl: Move HDM decoder helpers to built-in code Srirangan Madhavan
2026-11-05 19:07 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 07/19] cxl: Share HDM decoder register unpacking Srirangan Madhavan
2026-11-05 19:16 ` sashiko-bot
2026-11-12 21:35 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 08/19] cxl: Move cxl_rwsem into built-in code Srirangan Madhavan
2026-11-05 19:09 ` sashiko-bot
2026-11-12 23:49 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 09/19] cxl: Refresh cached PCI HDM decoder settings Srirangan Madhavan
2026-11-05 19:17 ` sashiko-bot
2026-11-13 0:38 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 10/19] cxl: Build register block helpers into the kernel Srirangan Madhavan
2026-11-05 19:23 ` sashiko-bot
2026-11-13 18:02 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 11/19] PCI/CXL: Cache endpoint HDM state during PCI enumeration Srirangan Madhavan
2026-11-05 19:30 ` sashiko-bot
2026-11-10 14:12 ` Li Ming
2026-11-11 3:54 ` Li Ming
2026-11-17 17:29 ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing Srirangan Madhavan
2026-11-05 19:17 ` sashiko-bot
2026-11-10 14:29 ` Li Ming
2026-11-11 10:12 ` Lucero Palau, Alejandro
2026-11-05 18:59 ` [PATCH v17 13/19] cxl: Validate and synchronize HDM ranges around reset Srirangan Madhavan
2026-11-05 19:44 ` sashiko-bot
2026-11-17 17:32 ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 14/19] PCI/CXL: Reject reset with unsafe function scope Srirangan Madhavan
2026-11-05 19:33 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 15/19] PCI/CXL: Restore CXL state after PCI reset Srirangan Madhavan
2026-11-05 19:34 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 16/19] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-11-05 19:35 ` sashiko-bot
2026-11-17 17:37 ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 17/19] PCI/CXL: Restore CXL state after CXL bus reset Srirangan Madhavan
2026-11-05 19:43 ` sashiko-bot
2026-11-13 17:37 ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 18/19] PCI/CXL: Flush host and device caches for " Srirangan Madhavan
2026-11-05 19:56 ` sashiko-bot
2026-11-05 18:59 ` [PATCH v17 19/19] cxl: Add KUnit checks for HDM decoder unpacking Srirangan Madhavan
2026-11-05 19:38 ` sashiko-bot
2026-11-05 19:31 ` [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Alison Schofield
2026-11-16 19:11 ` Dave Jiang [this message]
2026-11-17 17:28 ` Alison Schofield
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5616c825-454e-4eba-9352-b06d98dc123a@intel.com \
--to=dave.jiang@intel.com \
--cc=alex.williamson@redhat.com \
--cc=alison.schofield@intel.com \
--cc=alwilliamson@nvidia.com \
--cc=bhelgaas@google.com \
--cc=dave@stgolabs.net \
--cc=icheng@nvidia.com \
--cc=iweiny@kernel.org \
--cc=jan@nvidia.com \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=machaujo@amazon.de \
--cc=mhonap@nvidia.com \
--cc=ming.li@zohomail.com \
--cc=skancherla@nvidia.com \
--cc=smadhavan@nvidia.com \
--cc=vaslot@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.