From: Dave Jiang <dave.jiang@intel.com>
To: Srirangan Madhavan <smadhavan@nvidia.com>,
Alison Schofield <alison.schofield@intel.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Dan Williams <djbw@kernel.org>,
Davidlohr Bueso <dave@stgolabs.net>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <jic23@kernel.org>,
Vishal Verma <vishal.l.verma@intel.com>,
linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: Alex Williamson <alex.williamson@redhat.com>,
vsethi@nvidia.com, alwilliamson@nvidia.com,
Dan Williams <danwilliams@nvidia.com>,
Sai Yashwanth Reddy Kancherla <skancherla@nvidia.com>,
Vishal Aslot <vaslot@nvidia.com>,
Manish Honap <mhonap@nvidia.com>, Jiandi An <jan@nvidia.com>,
Richard Cheng <icheng@nvidia.com>,
linux-tegra@vger.kernel.org
Subject: Re: [PATCH v8 05/10] cxl: Add CXL Device Reset helper
Date: Tue, 7 Jul 2026 12:14:20 -0700 [thread overview]
Message-ID: <bedf65ed-1fac-4d2f-b0de-ed3b5dec391e@intel.com> (raw)
In-Reply-To: <20260703220508.546528-6-smadhavan@nvidia.com>
On 7/3/26 3:05 PM, Srirangan Madhavan wrote:
> Add an internal CXL Device Reset helper for Type 2 functions that advertise
> CXL Reset in the CXL Device DVSEC. The helper disables CXL.cache, performs
> cache writeback when supported, initiates reset with Memory Clear disabled,
> waits for completion, and re-enables CXL.cache on exit.
>
> Leave the helper unregistered until range validation and reset-scope
> validation are in place.
>
> Signed-off-by: Srirangan Madhavan <smadhavan@nvidia.com>
> ---
> drivers/cxl/core/reset.c | 224 ++++++++++++++++++++++++++++++++++
> include/cxl/cxl.h | 7 ++
> include/uapi/linux/pci_regs.h | 14 +++
> 3 files changed, 245 insertions(+)
>
> diff --git a/drivers/cxl/core/reset.c b/drivers/cxl/core/reset.c
> index 97b72cc67b6b..7a9b1ecfadcf 100644
> --- a/drivers/cxl/core/reset.c
> +++ b/drivers/cxl/core/reset.c
> @@ -7,6 +7,8 @@
> #include <linux/export.h>
> #include <linux/io.h>
> #include <linux/ioport.h>
> +#include <linux/iommu.h>
> +#include <linux/jiffies.h>
> #include <linux/kernel.h>
> #include <linux/pci.h>
> #include <linux/slab.h>
> @@ -449,3 +451,225 @@ void pci_cxl_hdm_init(struct pci_dev *pdev)
> if (rc && rc != -ENOTTY && rc != -ENODEV)
> pci_dbg(pdev, "CXL HDM cache init failed: %d\n", rc);
> }
> +
> +/*
> + * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings.
> + * Sec 9.7.3 leaves config-space access behavior undefined for 100 ms after
> + * initiating CXL Reset, then limits software to CXL Status2 access until
> + * reset completion, timeout, or error.
> + */
> +#define CXL_RESET_RRS_WAIT_MS 100
> +#define CXL_RESET_STATUS_POLL_MS 20
> +static const u32 cxl_reset_timeout_ms[] = {
> + 10, 100, 1000, 10000, 100000,
> +};
> +
> +#define CXL_CACHE_WBI_TIMEOUT_US 100000
> +#define CXL_CACHE_WBI_POLL_US 100
> +
> +static int cxl_reset_dvsec(struct pci_dev *pdev)
> +{
> + int dvsec, rc;
> + u16 cap;
> +
> + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
> + PCI_DVSEC_CXL_DEVICE);
> + if (!dvsec)
> + return -ENOTTY;
> +
> + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
> + if (rc)
> + return pcibios_err_to_errno(rc);
> +
> + if ((cap & (PCI_DVSEC_CXL_CACHE_CAPABLE |
> + PCI_DVSEC_CXL_MEM_CAPABLE)) !=
> + (PCI_DVSEC_CXL_CACHE_CAPABLE | PCI_DVSEC_CXL_MEM_CAPABLE))
Maybe
if (!((cap & PCI_DVSEC_CXL_CACHE_CAPABLE) &&
(cap & PCI_DVSEC_CXL_MEM_CAPABLE)))
> + return -ENOTTY;
> +
> + if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE))
> + return -ENOTTY;
> +
> + return dvsec;
> +}
> +
> +static int cxl_reset_update_ctrl2(struct pci_dev *pdev, int dvsec, u16 set,
> + u16 clear)
Would prefer to split this into two separate helper functions instead of combining set and clear. Especially given below the calls are either/or in all instances.
> +{
> + u16 cmd = PCI_DVSEC_CXL_INIT_CACHE_WBI | PCI_DVSEC_CXL_INIT_CXL_RST;
> + u16 ctrl2;
> + int rc;
> +
> + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2);
> + if (rc)
> + return pcibios_err_to_errno(rc);
> +
> + ctrl2 &= ~cmd;
> + ctrl2 |= set;
> + ctrl2 &= ~clear;
> +
> + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2);
> + if (rc)
> + return pcibios_err_to_errno(rc);
> +
> + return 0;
> +}
> +
> +static int cxl_reset_enable_cache(struct pci_dev *pdev, int dvsec)
> +{
> + return cxl_reset_update_ctrl2(pdev, dvsec, 0,
> + PCI_DVSEC_CXL_DISABLE_CACHING);
> +}
> +
> +static int cxl_reset_disable_cache(struct pci_dev *pdev, int dvsec, u16 cap)
> +{
> + int remaining_us = CXL_CACHE_WBI_TIMEOUT_US;
> + u16 status2;
> + int rc, rc2;
> +
> + rc = cxl_reset_update_ctrl2(pdev, dvsec,
> + PCI_DVSEC_CXL_DISABLE_CACHING, 0);
> + if (rc)
> + return rc;
> +
> + if (!(cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE))
> + return 0;
> +
> + rc = cxl_reset_update_ctrl2(pdev, dvsec,
> + PCI_DVSEC_CXL_INIT_CACHE_WBI, 0);
> + if (rc)
> + goto err_enable_cache;
> +
> + do {
> + usleep_range(CXL_CACHE_WBI_POLL_US, CXL_CACHE_WBI_POLL_US + 1);
> + remaining_us -= CXL_CACHE_WBI_POLL_US;
> +
> + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
> + &status2);
> + if (rc) {
> + rc = pcibios_err_to_errno(rc);
> + goto err_enable_cache;
> + }
> + } while (!(status2 & PCI_DVSEC_CXL_CACHE_INV) && remaining_us > 0);
> +
> + if (!(status2 & PCI_DVSEC_CXL_CACHE_INV)) {
> + rc = -ETIMEDOUT;
> + goto err_enable_cache;
> + }
> +
> + return 0;
> +
> +err_enable_cache:
> + /*
> + * DISABLE_CACHING can be rolled back here. INIT_CACHE_WBI is
> + * self-clearing on completion, so leave any in-flight writeback alone.
> + */
> + rc2 = cxl_reset_enable_cache(pdev, dvsec);
> + if (rc2)
> + pci_warn(pdev, "failed to re-enable CXL caching: %d\n", rc2);
> + return rc;
> +}
This may be more readable:
static int cxl_reset_wait_cache_wbi(struct pci_dev *pdev, int dvsec)
{
int remaining_us = CXL_CACHE_WBI_TIMEOUT_US;
u16 status2;
int rc;
rc = cxl_reset_update_ctrl2(pdev, dvsec,
PCI_DVSEC_CXL_INIT_CACHE_WBI, 0);
if (rc)
return rc;
do {
usleep_range(CXL_CACHE_WBI_POLL_US, CXL_CACHE_WBI_POLL_US + 1);
remaining_us -= CXL_CACHE_WBI_POLL_US;
rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
&status2);
if (rc)
return pcibios_err_to_errno(rc);
} while (!(status2 & PCI_DVSEC_CXL_CACHE_INV) && remaining_us > 0);
if (!(status2 & PCI_DVSEC_CXL_CACHE_INV))
return -ETIMEDOUT;
return 0;
}
static int cxl_reset_disable_cache(struct pci_dev *pdev, int dvsec, u16 cap)
{
int rc, rc2;
rc = cxl_reset_update_ctrl2(pdev, dvsec,
PCI_DVSEC_CXL_DISABLE_CACHING, 0);
if (rc)
return rc;
if (!(cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE))
return 0;
rc = cxl_reset_wait_cache_wbi(pdev, dvsec);
if (rc) {
rc2 = cxl_reset_enable_cache(pdev, dvsec);
if (rc2)
pci_warn(pdev, "failed to re-enable CXL caching: %d\n",
rc2);
}
return rc;
}
> +
> +static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap)
> +{
> + unsigned long deadline;
> + u32 timeout_ms;
> + u16 status2;
> + int idx, rc;
> +
> + idx = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, cap);
> + if (idx >= ARRAY_SIZE(cxl_reset_timeout_ms)) {
> + int last = ARRAY_SIZE(cxl_reset_timeout_ms) - 1;
> +
> + pci_warn(pdev,
> + "unknown CXL reset timeout encoding %d; using %u ms\n",
> + idx, cxl_reset_timeout_ms[last]);
> + idx = last;
> + }
> +
> + timeout_ms = max_t(u32, cxl_reset_timeout_ms[idx],
> + CXL_RESET_RRS_WAIT_MS);
> + deadline = jiffies + msecs_to_jiffies(timeout_ms);
> + msleep(CXL_RESET_RRS_WAIT_MS);
> +
> + do {
> + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
> + &status2);
> + if (rc || status2 == U16_MAX)
> + goto not_ready;
> +
> + if (status2 & PCI_DVSEC_CXL_RST_ERR)
> + return -EIO;
> +
> + if (status2 & PCI_DVSEC_CXL_RST_DONE)
> + return 0;
> +
> +not_ready:
> + if (time_after_eq(jiffies, deadline))
> + return -ETIMEDOUT;
> +
> + msleep(CXL_RESET_STATUS_POLL_MS);
> + } while (true);
You can skip the goto label:
do {
rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2,
&status2);
if (!rc && status2 != U16_MAX) {
if (status2 & PCI_DVSEC_CXL_RST_ERR)
return -EIO;
if (status2 & PCI_DVSEC_CXL_RST_DONE)
return 0;
}
if (time_after_eq(jiffies, deadline))
return -ETIMEDOUT;
msleep(CXL_RESET_STATUS_POLL_MS);
} while (true);
> +}
> +
> +static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
> +{
> + bool cache_disabled = false;
> + u16 cap;
> + int rc;
> +
> + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
> + if (rc)
> + return pcibios_err_to_errno(rc);
> +
> + if (!pci_wait_for_pending_transaction(pdev))
> + pci_err(pdev, "timed out waiting for pending transactions\n");
> +
> + rc = pci_dev_reset_iommu_prepare(pdev);
> + if (rc) {
> + pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
> + return rc;
> + }
> +
> + rc = cxl_reset_disable_cache(pdev, dvsec, cap);
> + if (rc)
> + goto out;
> + cache_disabled = true;
> +
> + rc = cxl_reset_update_ctrl2(pdev, dvsec, PCI_DVSEC_CXL_INIT_CXL_RST,
> + PCI_DVSEC_CXL_RST_MEM_CLR_EN);
> + if (rc)
> + goto out;
> +
> + rc = cxl_reset_wait_done(pdev, dvsec, cap);
> + if (rc)
> + goto out;
> +
> +out:
> + if (cache_disabled) {
> + int rc2;
> +
> + rc2 = cxl_reset_enable_cache(pdev, dvsec);
> + if (rc2 && rc)
> + pci_warn(pdev, "failed to re-enable CXL caching: %d\n",
> + rc2);
> + else if (rc2)
> + rc = rc2;
> + }
> +
> + pci_dev_reset_iommu_done(pdev);
> + return rc;
> +}
Can skip the cache_disabled and goto:
static int cxl_reset_execute(struct pci_dev *pdev, int dvsec)
{
u16 cap;
int rc;
rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
if (rc)
return pcibios_err_to_errno(rc);
if (!pci_wait_for_pending_transaction(pdev))
pci_err(pdev, "timed out waiting for pending transactions\n");
rc = pci_dev_reset_iommu_prepare(pdev);
if (rc) {
pci_err(pdev, "failed to stop IOMMU for CXL reset: %d\n", rc);
return rc;
}
rc = cxl_reset_disable_cache(pdev, dvsec, cap);
if (!rc) {
int rc2;
rc = cxl_reset_update_ctrl2(pdev, dvsec,
PCI_DVSEC_CXL_INIT_CXL_RST,
PCI_DVSEC_CXL_RST_MEM_CLR_EN);
if (!rc)
rc = cxl_reset_wait_done(pdev, dvsec, cap);
rc2 = cxl_reset_enable_cache(pdev, dvsec);
if (rc2 && rc)
pci_warn(pdev, "failed to re-enable CXL caching: %d\n",
rc2);
else if (rc2)
rc = rc2;
}
pci_dev_reset_iommu_done(pdev);
return rc;
}
DJ
> +
> +int cxl_reset_function(struct pci_dev *pdev, bool probe)
> +{
> + int dvsec;
> +
> + dvsec = cxl_reset_dvsec(pdev);
> + if (dvsec < 0)
> + return dvsec;
> +
> + if (probe)
> + return 0;
> +
> + return cxl_reset_execute(pdev, dvsec);
> +}
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index 2215fe1c3f78..de58f484b7d9 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -9,6 +9,7 @@
> #include <linux/node.h>
> #include <linux/ioport.h>
> #include <linux/range.h>
> +#include <linux/errno.h>
> #include <cxl/mailbox.h>
>
> /**
> @@ -153,6 +154,7 @@ int cxl_commit(struct cxl_decoder_settings *settings, void __iomem *hdm);
> #ifdef CONFIG_CXL_HDM
> void pci_cxl_hdm_init(struct pci_dev *pdev);
> void pci_cxl_hdm_release(struct pci_dev *pdev);
> +int cxl_reset_function(struct pci_dev *pdev, bool probe);
> #else
> static inline void pci_cxl_hdm_init(struct pci_dev *pdev)
> {
> @@ -161,6 +163,11 @@ static inline void pci_cxl_hdm_init(struct pci_dev *pdev)
> static inline void pci_cxl_hdm_release(struct pci_dev *pdev)
> {
> }
> +
> +static inline int cxl_reset_function(struct pci_dev *pdev, bool probe)
> +{
> + return -ENOTTY;
> +}
> #endif
>
> struct cxl_reg_map {
> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> index 14f634ab9350..194ae56b4404 100644
> --- a/include/uapi/linux/pci_regs.h
> +++ b/include/uapi/linux/pci_regs.h
> @@ -1349,10 +1349,24 @@
> /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */
> #define PCI_DVSEC_CXL_DEVICE 0
> #define PCI_DVSEC_CXL_CAP 0xA
> +#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0)
> #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2)
> #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4)
> +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6)
> +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7)
> +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8)
> +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11)
> #define PCI_DVSEC_CXL_CTRL 0xC
> #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2)
> +#define PCI_DVSEC_CXL_CTRL2 0x10
> +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0)
> +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1)
> +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2)
> +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3)
> +#define PCI_DVSEC_CXL_STATUS2 0x12
> +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0)
> +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1)
> +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2)
> #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10))
> #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10))
> #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0)
next prev parent reply other threads:[~2026-07-07 19:14 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-03 22:04 [PATCH v8 00/10] PCI/CXL: Add CXL reset support for Type 2 devices Srirangan Madhavan
2026-07-03 22:04 ` [PATCH v8 01/10] cxl: Split decoder programming into a reusable helper Srirangan Madhavan
2026-07-07 0:22 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 02/10] cxl: Cache decoder settings on PCI devices Srirangan Madhavan
2026-07-07 16:45 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 03/10] cxl: Share HDM decoder decode logic Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 04/10] cxl: Cache endpoint decoder settings during PCI enumeration Srirangan Madhavan
2026-07-07 5:30 ` Richard Cheng
2026-07-07 17:47 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 05/10] cxl: Add CXL Device Reset helper Srirangan Madhavan
2026-07-07 19:14 ` Dave Jiang [this message]
2026-07-03 22:05 ` [PATCH v8 06/10] cxl: Validate HDM ranges before CXL reset Srirangan Madhavan
2026-07-07 21:34 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 07/10] PCI/CXL: Discover the CXL reset scope Srirangan Madhavan
2026-07-07 1:45 ` Richard Cheng
2026-07-03 22:05 ` [PATCH v8 08/10] cxl: Restore CXL HDM state after PCI reset Srirangan Madhavan
2026-07-07 1:34 ` Richard Cheng
2026-07-07 22:30 ` Dave Jiang
2026-07-03 22:05 ` [PATCH v8 09/10] PCI/CXL: Expose CXL Reset as a PCI reset method Srirangan Madhavan
2026-07-03 22:05 ` [PATCH v8 10/10] Documentation/ABI: Document CXL Reset " Srirangan Madhavan
2026-07-03 22:08 ` [PATCH RFC] PCI/CXL: Restore HDM state after CXL bus reset Srirangan Madhavan
2026-07-06 23:13 ` [PATCH v8 00/10] PCI/CXL: Add CXL reset support for Type 2 devices Dave Jiang
2026-07-07 22:45 ` Dave Jiang
2026-07-07 5:41 ` Richard Cheng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bedf65ed-1fac-4d2f-b0de-ed3b5dec391e@intel.com \
--to=dave.jiang@intel.com \
--cc=alex.williamson@redhat.com \
--cc=alison.schofield@intel.com \
--cc=alwilliamson@nvidia.com \
--cc=bhelgaas@google.com \
--cc=danwilliams@nvidia.com \
--cc=dave@stgolabs.net \
--cc=djbw@kernel.org \
--cc=icheng@nvidia.com \
--cc=ira.weiny@intel.com \
--cc=jan@nvidia.com \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mhonap@nvidia.com \
--cc=skancherla@nvidia.com \
--cc=smadhavan@nvidia.com \
--cc=vaslot@nvidia.com \
--cc=vishal.l.verma@intel.com \
--cc=vsethi@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox