[PATCH v6 29/31] vfio/cxl: Run the CXL reset at the vfio reset points

[<mhonap@nvidia.com>]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 function must not take an FLR in place of CXL Reset: an FLR
leaves CXL.cache and CXL.mem state, the CXL DVSEC and the HDM contents
untouched (CXL r4.0 sec 9.5), so device memory and cache would carry one
user's data to the next. CXL Reset resets that state as well (sec 9.7).
The PCI core orders cxl_reset ahead of flr in pci_reset_fn_methods[],
but falls through to FLR when cxl_reset returns -ENOTTY.

vfio-pci-core routes its reset points through the provider reset op. In
vfio-cxl that op runs pci_cxl_reset_owned() with the HDM range claim.
The reset resets the function, clearing or randomizing volatile device
memory, and restores the HDM decoder and PCI config state, so it is a
complete replacement for pci_try_reset_function() on a CXL device. The
op runs with memory_lock and the PCI device lock held.

Take the HDM memory region out of use for the duration of the reset, so
a fault cannot insert a PFN into a decoder that is being torn down, and
put it back only once the reset has restored the decoder and the device
reports its memory active again. After CXL Reset with Memory Clear the
device can keep Memory_Active clear for seconds (about 3 s on the device
this was tested on), and a host access to the HDM range in that window
raised an SError. A reset whose memory does not come back within the
Memory_Active_Timeout fails like any other. A failed reset
returns its error and leaves the region out of use until a later CXL
Reset succeeds. Neither a decoder restore after low power nor a hot
reset reopens it in between, since device memory may still hold data the
reset was meant to clear. If a failed reset left CXL.cache or CXL.mem
disabled in the DVSEC Control register, which makes cxl_reset refuse the
device, put back the Control value taken at bind before the next reset.
A failed reset leaves Memory Space disabled; clear vfio-pci-core's
record of it to match. AER recovery does not reopen HDM after a failed
CXL Reset either.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 82 ++++++++++++++++++++++++++--
 1 file changed, 77 insertions(+), 5 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 518f0b093652..70794efd517b 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -38,6 +38,8 @@
  *	for Memory_Active before host access reopens after a reset or low power
  * @mem_region_index: vfio region index of the HDM memory region while open, else 0
  * @hdm_open_at_prepare: @hdm_valid as reset_prepare found it; under memory_lock
+ * @hdm_needs_reset: a CXL Reset failed, so only a CXL Reset that succeeds
+ *	reopens host access; under memory_lock
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
@@ -52,6 +54,7 @@ struct vfio_cxl_state {
 	bool await_mem_active;
 	unsigned int mem_region_index;
 	bool hdm_open_at_prepare;
+	bool hdm_needs_reset;
 };
 
 static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma,
@@ -827,17 +830,75 @@ static void vfio_cxl_sync_mem_enable(struct vfio_pci_core_device *vdev)
 }
 
 /*
- * vfio-cxl holds the HDM range, so the PCI core's cxl_reset method would
- * fail to request it. Run the same CXL Reset with that claim.
+ * cxl_reset refuses a device whose CXL DVSEC Control has Cache_Enable or
+ * Mem_Enable clear (-ENOTTY). A CXL Reset that failed after the device reset
+ * may leave Control at its reset value, so every later reset would refuse and
+ * host access would never reopen. Put back the Control value taken at bind,
+ * when pci_cxl_reset_capable() required both enables, before the reset.
+ */
+static void vfio_cxl_restore_dvsec_ctrl(struct vfio_pci_core_device *vdev)
+{
+	const u16 enables = PCI_DVSEC_CXL_CACHE_ENABLE | PCI_DVSEC_CXL_MEM_ENABLE;
+	const struct cxl_hdm_info *hdm = vdev->cxl->hdm;
+	struct pci_dev *pdev = vdev->pdev;
+	u16 ctrl;
+
+	if (!hdm->dvsec_ctrl_valid || (hdm->dvsec_ctrl & enables) != enables)
+		return;
+
+	if (pci_read_config_word(pdev, hdm->dvsec + PCI_DVSEC_CXL_CTRL, &ctrl) ||
+	    PCI_POSSIBLE_ERROR(ctrl) || (ctrl & enables) == enables)
+		return;
+
+	pci_warn(pdev, "vfio-cxl: restoring CXL DVSEC control %#06x (was %#06x)\n",
+		 hdm->dvsec_ctrl, ctrl);
+	pci_write_config_word(pdev, hdm->dvsec + PCI_DVSEC_CXL_CTRL,
+			      hdm->dvsec_ctrl);
+}
+
+/*
+ * Run CXL Reset in place of a PCI function reset. An FLR leaves CXL.mem
+ * state and HDM contents intact (CXL r4.0 sec 9.5), so the vfio reset points
+ * route here. The reset clears or randomizes volatile device memory and
+ * restores the HDM decoder. The caller holds memory_lock and the PCI device
+ * lock. vfio-cxl holds the HDM range, so the PCI core's cxl_reset method
+ * would fail to request it; run the same CXL Reset with that claim.
  */
 static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	/* Host CPU access to the HDM range is unsafe until the decoder is back. */
+	vfio_cxl_revoke_hdm(vdev);
+
+	/* Only after a failure: see vfio_cxl_restore_dvsec_ctrl(). */
+	if (vdev->cxl->hdm_needs_reset)
+		vfio_cxl_restore_dvsec_ctrl(vdev);
+
 	/*
 	 * The reset restores PCI state from the saved copy. The caller provides
 	 * it: the try-reset helper saves the current state, and the close path
 	 * keeps the open-time state that vfio_pci_core_disable() loaded.
 	 */
-	return pci_cxl_reset_owned(vdev->pdev, vdev->cxl->hpa_res);
+	ret = pci_cxl_reset_owned(pdev, vdev->cxl->hpa_res);
+	if (!ret)
+		ret = vfio_cxl_await_mem_active(vdev);
+	if (ret) {
+		/*
+		 * Device memory may hold data the reset was meant to clear. Only
+		 * a later CXL Reset that succeeds reopens host access; a decoder
+		 * restore alone does not.
+		 */
+		vdev->cxl->hdm_needs_reset = true;
+		vfio_cxl_sync_mem_enable(vdev);
+		return ret;
+	}
+
+	vdev->cxl->hdm_needs_reset = false;
+	vfio_cxl_rearm_hdm(vdev);
+
+	return 0;
 }
 
 /*
@@ -881,6 +942,10 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 
 	lockdep_assert_held_write(&vdev->memory_lock);
 
+	/* After a failed CXL Reset, only a CXL Reset reopens host access. */
+	if (cxl->hdm_needs_reset)
+		return;
+
 	/*
 	 * A D3 request that failed, or a quirk that kept the device in D0,
 	 * lost nothing: reopen without decommitting and recommitting.
@@ -984,6 +1049,10 @@ static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
 		return 0;
 	}
 
+	/* After a failed CXL Reset, only a CXL Reset reopens host access. */
+	if (cxl->hdm_needs_reset)
+		return 0;
+
 	ret = vfio_cxl_await_mem_active(vdev);
 	if (ret)
 		return ret;
@@ -1007,8 +1076,11 @@ static void vfio_cxl_error_resume(struct vfio_pci_core_device *vdev)
 	if (pdev->current_state != PCI_D0)
 		return;
 
-	/* The region was closed before the error; keep it that way. */
-	if (!vdev->cxl->hdm_open_at_prepare)
+	/*
+	 * The region was closed before the error, or a CXL Reset failed while
+	 * recovery ran; keep it closed until a CXL Reset succeeds.
+	 */
+	if (!vdev->cxl->hdm_open_at_prepare || vdev->cxl->hdm_needs_reset)
 		return;
 
 	if (!vfio_cxl_hdm_intact(vdev)) {
-- 
2.53.0