Linux Tegra architecture development
 help / color / mirror / Atom feed
From: Srirangan Madhavan <smadhavan@nvidia.com>
To: Dave Jiang <dave.jiang@intel.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Ira Weiny <iweiny@kernel.org>,
	Jonathan Cameron <jic23@kernel.org>,
	Vishal Verma <vishal.l.verma@intel.com>,
	linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
	linux-kernel@vger.kernel.org
Cc: Li Ming <ming.li@zohomail.com>,
	Joshua Machauer <machaujo@amazon.de>,
	Alex Williamson <alex.williamson@redhat.com>,
	vsethi@nvidia.com, alwilliamson@nvidia.com,
	Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
	Vishal Aslot <vaslot@nvidia.com>,
	Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
	Richard Cheng <icheng@nvidia.com>,
	linux-tegra@vger.kernel.org
Subject: Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices
Date: Thu, 19 Nov 2026 23:32:08 -0800	[thread overview]
Message-ID: <b35bcc93-9fb0-4fa2-a432-857eaa4dd31b@nvidia.com> (raw)
In-Reply-To: <5616c825-454e-4eba-9352-b06d98dc123a@intel.com>

On 11/16/26 11:11 AM, Dave Jiang wrote:
> External email: Use caution opening links or attachments
> 
> 
> On 11/5/26 11:58 AM, Srirangan Madhavan wrote:
>> Hi folks,
>>
>> This series adds CXL Reset for Type 2 devices through the existing PCI
>> reset_method and reset sysfs attributes. It disables CXL.cache, performs
>> supported cache writeback, initiates CXL Reset with Memory Clear, and
>> restores cached HDM decoder and CXL Device DVSEC state before PCI use
>> resumes. Active HDM ranges are reserved and CPU caches invalidated before
>> and after reset. The method remains limited to eligible single-function
>> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
>> the same cached state and flushes the host and device caches around the
>> reset.
>>
>> The reset sequence follows the CXL 4.0 specification [4], including
>> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
>> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
>> define the completion timeout encodings and reset access restrictions.
>>
>> The userspace ABI is the existing PCI reset interface:
>>
>>      /sys/bus/pci/devices/.../reset_method
>>      /sys/bus/pci/devices/.../reset
>>
>> Userspace selects "cxl_reset" in reset_method and triggers the reset
>> through reset. Following Dan's v6 feedback, this uses the PCI interface
>> instead of adding a memdev-specific sysfs ABI.
>>
>> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
>> callback when provided and disables PCI I/O, memory decoding, and bus
>> mastering. The CXL path reserves affected HDM ranges exclusively and
>> blocks IOMMU translations where applicable. It does not offline memory
>> or revoke mappings: userspace is expected to offline CXL.mem users before
>> reset. The reservation enforces this for claimed ranges by failing the
>> reset while a conflicting resource is claimed [9].
>>
>> The global region lock is held from snapshot through restoration. Memory
>> Clear completion is awaited for the timeout the device advertises, up to
>> 100 seconds, so the lock can be held for that long plus cache flushes.
> 
> I think holding it across the bus reset can deadlock with pciehp and a
> PCI remove. I tested in QEMU with two cxl-type2 devices from Manish's
> RFC [1].
> 
> For a device in a pciehp slot, pci_reset_bus_function() ends up in
> pciehp_reset_slot(), which takes the slot's reset_lock. The reset still
> holds cxl_rwsem.region at that point.
> 
> Two other tasks can complete a cycle. pciehp_ist() for that slot can
> hold reset_lock and wait for pci_rescan_remove_lock in
> pciehp_unconfigure_device(). A PCI remove of a second Type 2 device can
> hold pci_rescan_remove_lock while cxl_decoder_detach() waits for
> cxl_rwsem.region.
> 
> On v17 as posted, lockdep reports the cycle:
> 
>    [   60.891789] WARNING: possible circular locking dependency detected
>    [   60.893014] irq/24-pciehp/103 is trying to acquire lock:
>    [   60.893367] ffff888072b4f128 (&cxl_memdev_key){+.+.}-{4:4}, at: device_del+0x8d/0x580
>                   but task is already holding lock:
>    [   60.894498] ffffffff84699e80 (pci_rescan_remove_lock){+.+.}-{4:4}, at: pciehp_unconfigure_device+0x93/0x210
>    [...]
>                   -> #3 (&ctrl->reset_lock){++++}-{4:4}:
>    [   60.916794]        down_write_nested+0x80/0x170
>    [   60.917826]        pciehp_reset_slot+0x6c/0xc0
>    [   60.918893]        pci_reset_hotplug_slot+0x5c/0x90
>    [   60.919928]        pci_reset_bus_function+0x161/0x330
>    [   60.920951]        cxl_reset_bus_function+0x1c6/0x280
>    [   60.921958]        __pci_reset_function_locked+0x96/0x170
>    [   60.922990]        pci_reset_function+0xc8/0x180
>    [   60.923977]        reset_store+0x84/0xd0
>    [...]
>                   -> #2 (cxl_rwsem.region){++++}-{4:4}:
>    [...]
>    [   61.013490] Chain exists of:
>                     &cxl_memdev_key --> &ctrl->reset_lock --> pci_rescan_remove_lock
> 
> With a test-only msleep() after cxl_region_quiesce_lock() in
> pci_cxl_reset_prepare(), it turns into a real hang. The reset, a sysfs
> remove of the second device and a power off of the first device's slot
> all got stuck.
> 
> Two region reads blocked behind the reset as well. A read of
> region1/commit timed out after 5 s, and cxl list -R stayed in D state in
> decoders_committed_show().
> 
> To reproduce, I used Manish's RFC as posted. It allows one cxl-type2 per
> pxb-cxl, so each device has its own host bridge and window. On q35 with
> cxl=on, plus the memory backends:
> 
>    -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=53
>    -device cxl-rp,id=hb0rp0,bus=cxl.0,chassis=0,slot=0,port=0
>    -device cxl-type2,bus=hb0rp0,volatile-memdev=t2mem0,id=cxl-t2,fmw-index=0
>    -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191
>    -device cxl-rp,id=hb1rp0,bus=cxl.1,chassis=0,slot=2,port=0
>    -device cxl-type2,bus=hb1rp0,volatile-memdev=t2mem1,id=cxl-t2b,fmw-index=1
>    -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G
> 
> The guest kernel needs CONFIG_CXL_REGION_INVALIDATION_TEST=y. Without it,
> the cxl_bus reset fails in an x86 guest with "failed to prepare CXL bus
> reset ranges: -6".
> 
> I drove both devices with a small test module based on
> tools/testing/cxl/test/accel.c. It attaches each firmware-committed
> region with devm_cxl_probe_mem() and has no reset handlers. Then, with
> lockdep on:
> 
>    echo cxl_bus > /sys/bus/pci/devices/0000:36:00.0/reset_method
>    echo 1 > /sys/bus/pci/devices/0000:36:00.0/reset
>    echo 0 > /sys/bus/pci/slots/2/power     # second device, 0000:c0:00.0
> 
> For the hang, add msleep(10000) after cxl_region_quiesce_lock() in
> pci_cxl_reset_prepare(). Start the cxl_bus reset, remove 0000:c0:00.0
> through sysfs two seconds later, and power off slot 0 two seconds after
> that.
> 
> Separately, pci_cxl_reset_prepare() returns with the quiesce held on its
> success path only, and cxl_restore_state_after_pci_reset() releases it.
> With Clang 23 and CONTEXT_ANALYSIS=y passed to make, that gives five
> cxl_region_quiesce warnings in drivers/pci/cxl.c.
> 
> [1]: https://lore.kernel.org/all/20261106035911.1280066-1-mhonap@nvidia.com/
> 
> DJ

Hi Dave,

Thank you for the repro. Please let me know your opinion on the following fix.
It also covers Alison's lock scope question.

Reordering the locks won't work, since pci_reset_bus_function() itself
takes reset_lock. So I'd like to drop the region lock before the reset starts,
and take it only for two short windows:

   pci_cxl_reset_prepare()
       lock(region):  cxl_hdm_cache_snapshot(), set reset_pending,
                      cxl_hdm_ranges_collect() + cxl_hdm_ranges_request()
       cxl_hdm_quiesce_flush()                      [no lock]
   pci_reset_bus_function()                         [no region lock]
   cxl_restore_state_after_pci_reset()
       lock(region):  cxl_hdm_cache_copy_decoders(), pci_cxl_restore_state(),
                      clear reset_pending
       cxl_hdm_quiesce_flush()                      [no lock]

   cxl_decoder_commit():  reset_pending -> -EBUSY   [next to sanitize_active]
   disable_hdm():         also update the cached global_ctrl
   cxl_reset_execute():   same two windows for the DVSEC reset

With this,
  - If a region gets torn down mid-reset, the cache is already refreshed
    (cxl_hdm_cache_refresh_decoder()), so restore just re-reads it with
    cxl_hdm_cache_copy_decoders(). Patch 17 had that call, and patch 18
    dropped it when I moved the lock into prepare.
  - A new commit mid-reset gets -EBUSY. Regions committed before the reset
    shouldn't be able to come online, since the exclusive
    request_mem_region() makes the kmem and pmem claims fail, as far as I
    can tell.

Questions
  1. Is reset_pending in struct cxl_hdm_info, checked in
     cxl_decoder_commit(), OK?
  2. Can I keep cxl_hdm_cache_copy_decoders(), now used again, instead of
     dropping it?
  3. cxl_hdm_ranges_invalidate() and cxl_hdm_quiesce_prepare() assert the
     region lock is held. To flush outside it I'd relax that and rely on
     the HPA reservation. OK?

If this plan sounds okay, I can implement it this way in v18 (and rerun
your reproducer against it)

-- 
Regards,
Srirangan

  reply	other threads:[~2026-11-20  7:34 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-05 18:58 [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Srirangan Madhavan
2026-11-05 18:58 ` [PATCH v17 01/19] cxl: Clarify decoder interleave limit Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 02/19] cxl: Share CXL port upstream PCI device lookup Srirangan Madhavan
2026-11-10 13:36   ` Li Ming
2026-11-13 15:50     ` Dave Jiang
2026-11-15  3:25       ` Li Ming
2026-11-05 18:59 ` [PATCH v17 03/19] cxl: Move decoder declarations to shared header Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 04/19] cxl: Embed decoder configuration in a standalone structure Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 05/19] cxl: Introduce endpoint HDM decoder settings Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 06/19] cxl: Move HDM decoder helpers to built-in code Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 07/19] cxl: Share HDM decoder register unpacking Srirangan Madhavan
2026-11-12 21:35   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 08/19] cxl: Move cxl_rwsem into built-in code Srirangan Madhavan
2026-11-12 23:49   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 09/19] cxl: Refresh cached PCI HDM decoder settings Srirangan Madhavan
2026-11-13  0:38   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 10/19] cxl: Build register block helpers into the kernel Srirangan Madhavan
2026-11-13 18:02   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 11/19] PCI/CXL: Cache endpoint HDM state during PCI enumeration Srirangan Madhavan
2026-11-10 14:12   ` Li Ming
2026-11-11  3:54     ` Li Ming
2026-11-17 17:29   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 12/19] PCI/CXL: Add CXL Device Reset sequencing Srirangan Madhavan
2026-11-10 14:29   ` Li Ming
2026-11-11 10:12   ` Lucero Palau, Alejandro
2026-11-05 18:59 ` [PATCH v17 13/19] cxl: Validate and synchronize HDM ranges around reset Srirangan Madhavan
2026-11-17 17:32   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 14/19] PCI/CXL: Reject reset with unsafe function scope Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 15/19] PCI/CXL: Restore CXL state after PCI reset Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 16/19] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-11-17 17:37   ` Alison Schofield
2026-11-05 18:59 ` [PATCH v17 17/19] PCI/CXL: Restore CXL state after CXL bus reset Srirangan Madhavan
2026-11-13 17:37   ` Dave Jiang
2026-11-05 18:59 ` [PATCH v17 18/19] PCI/CXL: Flush host and device caches for " Srirangan Madhavan
2026-11-05 18:59 ` [PATCH v17 19/19] cxl: Add KUnit checks for HDM decoder unpacking Srirangan Madhavan
2026-11-05 19:31 ` [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices Alison Schofield
2026-11-16 19:11 ` Dave Jiang
2026-11-20  7:32   ` Srirangan Madhavan [this message]
2026-11-17 17:28 ` Alison Schofield

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b35bcc93-9fb0-4fa2-a432-857eaa4dd31b@nvidia.com \
    --to=smadhavan@nvidia.com \
    --cc=alex.williamson@redhat.com \
    --cc=alison.schofield@intel.com \
    --cc=alwilliamson@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=icheng@nvidia.com \
    --cc=iweiny@kernel.org \
    --cc=jan@nvidia.com \
    --cc=jic23@kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-tegra@vger.kernel.org \
    --cc=machaujo@amazon.de \
    --cc=mhonap@nvidia.com \
    --cc=ming.li@zohomail.com \
    --cc=skancherla@nvidia.com \
    --cc=vaslot@nvidia.com \
    --cc=vishal.l.verma@intel.com \
    --cc=vsethi@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox