[PATCH v2 0/2] Improve VF FLR synchronization for Xe VFIO

[PATCH v2 0/2] Improve VF FLR synchronization for Xe VFIO

[Piórkowski, Piotr]

From: Piotr Piórkowski <piotr.piorkowski@intel.com>

When xe-vfio-pci waits for VF FLR completion, it may start waiting
for "FLR done" before the PF driver is notified about the reset by
the GuC. With delayed HW/FW notification, the PF may not yet observe
VF FLR in progress when the wait path is entered, which can lead to
the reset being treated as completed before it has actually begun.

This series introduces an optional FLR_PREPARE state in the PF control
flow to mark that a VF reset is pending before the GuC VF FLR event is
received. The wait path treats this state the same as FLR_WIP, ensuring
that external waiters do not return prematurely.

The second patch hooks into the PCI error handler reset_prepare()
callback in xe-vfio-pci to notify the PF about the upcoming VF reset
before reset_done() is executed.

Piotr Piórkowski (2):
  drm/xe/pf: Add FLR_PREPARE state to VF control flow
  vfio/xe: Notify PF about VF FLR in reset_prepare

 drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c   | 78 +++++++++++++++----
 drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h   |  1 +
 .../gpu/drm/xe/xe_gt_sriov_pf_control_types.h |  2 +
 drivers/gpu/drm/xe/xe_sriov_pf_control.c      | 24 ++++++
 drivers/gpu/drm/xe/xe_sriov_pf_control.h      |  1 +
 drivers/gpu/drm/xe/xe_sriov_vfio.c            |  1 +
 drivers/vfio/pci/xe/main.c                    | 14 ++++
 include/drm/intel/xe_sriov_vfio.h             | 11 +++
 8 files changed, 116 insertions(+), 16 deletions(-)

-- 
2.34.1

[PATCH v2 1/2] drm/xe/pf: Add FLR_PREPARE state to VF control flow

[Piórkowski, Piotr]

From: Piotr Piórkowski <piotr.piorkowski@intel.com>

Our xe-vfio-pci component relies on the confirmation from the PF
that VF FLR processing has finished, but due to the notification
latency on the HW/FW side, PF might be unaware yet of the already
triggered VF FLR.

Update VF state machine with new FLR_PREPARE state that indicate
imminent VF FLR notification and treat that as a begin of the FLR
sequence. Also introduce function that xe-vfio-pci should call to
guarantee correct synchronization.

v2: move PREPARE into WIP, update commit msg (Michal)

Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Co-developed-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
---
 drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c   | 78 +++++++++++++++----
 drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h   |  1 +
 .../gpu/drm/xe/xe_gt_sriov_pf_control_types.h |  2 +
 drivers/gpu/drm/xe/xe_sriov_pf_control.c      | 24 ++++++
 drivers/gpu/drm/xe/xe_sriov_pf_control.h      |  1 +
 drivers/gpu/drm/xe/xe_sriov_vfio.c            |  1 +
 include/drm/intel/xe_sriov_vfio.h             | 11 +++
 7 files changed, 102 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
index 5cb705c7ee7a..058585f063a9 100644
--- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
+++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
@@ -171,6 +171,7 @@ static const char *control_bit_to_string(enum xe_gt_sriov_control_bits bit)
 	case XE_GT_SRIOV_STATE_##_X: return #_X
 	CASE2STR(WIP);
 	CASE2STR(FLR_WIP);
+	CASE2STR(FLR_PREPARE);
 	CASE2STR(FLR_SEND_START);
 	CASE2STR(FLR_WAIT_GUC);
 	CASE2STR(FLR_GUC_DONE);
@@ -1486,11 +1487,15 @@ int xe_gt_sriov_pf_control_stop_vf(struct xe_gt *gt, unsigned int vfid)
  * The VF FLR state machine looks like::
  *
  *	 (READY,PAUSED,STOPPED)<------------<--------------o
- *	    |                                               \
- *	   flr                                               \
- *	    |                                                 \
- *	....V..........................FLR_WIP...........      \
- *	:    \                                          :       \
+ *	    |             |                                 \
+ *	   flr           prepare                             \
+ *	    |             |                                   \
+ *	....V.............V............FLR_WIP...........      \
+ *	:   |             |                             :       \
+ *	:   |    FLR_PREPARE                            :        |
+ *	:   |    /                                      :        |
+ *	:   \   flr                                     :        |
+ *	:    \ /                                        :        |
  *	:     \   o----<----busy                        :        |
  *	:      \ /            /                         :        |
  *	:       FLR_SEND_START---failed----->-----------o--->(FLR_FAILED)<---o
@@ -1539,20 +1544,28 @@ static void pf_enter_vf_flr_send_start(struct xe_gt *gt, unsigned int vfid)
 	pf_queue_vf(gt, vfid);
 }
 
-static void pf_enter_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
+static bool pf_exit_vf_flr_prepare(struct xe_gt *gt, unsigned int vfid)
 {
-	if (!pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP)) {
-		xe_gt_sriov_dbg(gt, "VF%u FLR is already in progress\n", vfid);
-		return;
-	}
+	if (!pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE))
+		return false;
 
-	pf_enter_vf_wip(gt, vfid);
 	pf_enter_vf_flr_send_start(gt, vfid);
+	return true;
+}
+
+static bool pf_enter_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
+{
+	if (!pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP))
+		return false;
+
+	pf_enter_vf_wip(gt, vfid);
+	return true;
 }
 
 static void pf_exit_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
 {
 	if (pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP)) {
+		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE);
 		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_SEND_FINISH);
 		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_RESET_MMIO);
 		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_RESET_DATA);
@@ -1760,21 +1773,54 @@ static void pf_enter_vf_flr_guc_done(struct xe_gt *gt, unsigned int vfid)
 }
 
 /**
- * xe_gt_sriov_pf_control_trigger_flr - Start a VF FLR sequence.
+ * xe_gt_sriov_pf_control_prepare_flr() - Notify PF that VF FLR request was issued.
  * @gt: the &xe_gt
  * @vfid: the VF identifier
  *
+ * This is an optional early notification path used to mark pending FLR before
+ * the GuC notifies the PF with a FLR event.
+ *
  * This function is for PF only.
  *
  * Return: 0 on success or a negative error code on failure.
  */
-int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid)
+int xe_gt_sriov_pf_control_prepare_flr(struct xe_gt *gt, unsigned int vfid)
 {
-	pf_enter_vf_flr_wip(gt, vfid);
+	if (!pf_enter_vf_flr_wip(gt, vfid))
+		return -EALREADY;
 
+	pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE);
 	return 0;
 }
 
+static int pf_begin_vf_flr(struct xe_gt *gt, unsigned int vfid)
+{
+	if (pf_enter_vf_flr_wip(gt, vfid)) {
+		pf_enter_vf_flr_send_start(gt, vfid);
+		return 0;
+	}
+
+	if (pf_exit_vf_flr_prepare(gt, vfid))
+		return 0;
+
+	xe_gt_sriov_dbg(gt, "VF%u FLR is already in progress\n", vfid);
+	return -EALREADY;
+}
+
+/**
+ * xe_gt_sriov_pf_control_trigger_flr - Start a VF FLR sequence.
+ * @gt: the &xe_gt
+ * @vfid: the VF identifier
+ *
+ * This function is for PF only.
+ *
+ * Return: 0 on success or a negative error code on failure.
+ */
+int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid)
+{
+	return pf_begin_vf_flr(gt, vfid);
+}
+
 /**
  * xe_gt_sriov_pf_control_sync_flr() - Synchronize on the VF FLR checkpoint.
  * @gt: the &xe_gt
@@ -1879,9 +1925,9 @@ static void pf_handle_vf_flr(struct xe_gt *gt, u32 vfid)
 
 	if (needs_dispatch_flr(xe)) {
 		for_each_gt(gtit, xe, gtid)
-			pf_enter_vf_flr_wip(gtit, vfid);
+			pf_begin_vf_flr(gtit, vfid);
 	} else {
-		pf_enter_vf_flr_wip(gt, vfid);
+		pf_begin_vf_flr(gt, vfid);
 	}
 }
 
diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
index c36c8767f3ad..23182a5c5fb8 100644
--- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
+++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
@@ -27,6 +27,7 @@ int xe_gt_sriov_pf_control_process_restore_data(struct xe_gt *gt, unsigned int v
 int xe_gt_sriov_pf_control_trigger_restore_vf(struct xe_gt *gt, unsigned int vfid);
 int xe_gt_sriov_pf_control_finish_restore_vf(struct xe_gt *gt, unsigned int vfid);
 int xe_gt_sriov_pf_control_stop_vf(struct xe_gt *gt, unsigned int vfid);
+int xe_gt_sriov_pf_control_prepare_flr(struct xe_gt *gt, unsigned int vfid);
 int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid);
 int xe_gt_sriov_pf_control_sync_flr(struct xe_gt *gt, unsigned int vfid, bool sync);
 int xe_gt_sriov_pf_control_wait_flr(struct xe_gt *gt, unsigned int vfid);
diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
index 6027ba05a7f2..e78c59e08adf 100644
--- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
+++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
@@ -15,6 +15,7 @@
  *
  * @XE_GT_SRIOV_STATE_WIP: indicates that some operations are in progress.
  * @XE_GT_SRIOV_STATE_FLR_WIP: indicates that a VF FLR is in progress.
+ * @XE_GT_SRIOV_STATE_FLR_PREPARE: indicates that the PF received early VF FLR prepare notification.
  * @XE_GT_SRIOV_STATE_FLR_SEND_START: indicates that the PF wants to send a FLR START command.
  * @XE_GT_SRIOV_STATE_FLR_WAIT_GUC: indicates that the PF awaits for a response from the GuC.
  * @XE_GT_SRIOV_STATE_FLR_GUC_DONE: indicates that the PF has received a response from the GuC.
@@ -56,6 +57,7 @@ enum xe_gt_sriov_control_bits {
 	XE_GT_SRIOV_STATE_WIP = 1,
 
 	XE_GT_SRIOV_STATE_FLR_WIP,
+	XE_GT_SRIOV_STATE_FLR_PREPARE,
 	XE_GT_SRIOV_STATE_FLR_SEND_START,
 	XE_GT_SRIOV_STATE_FLR_WAIT_GUC,
 	XE_GT_SRIOV_STATE_FLR_GUC_DONE,
diff --git a/drivers/gpu/drm/xe/xe_sriov_pf_control.c b/drivers/gpu/drm/xe/xe_sriov_pf_control.c
index ed4b9820b06e..15b4341d7f12 100644
--- a/drivers/gpu/drm/xe/xe_sriov_pf_control.c
+++ b/drivers/gpu/drm/xe/xe_sriov_pf_control.c
@@ -123,6 +123,30 @@ int xe_sriov_pf_control_reset_vf(struct xe_device *xe, unsigned int vfid)
 	return result;
 }
 
+/**
+ * xe_sriov_pf_control_prepare_flr() - Notify PF that VF FLR prepare has started.
+ * @xe: the &xe_device
+ * @vfid: the VF identifier
+ *
+ * This function is for PF only.
+ *
+ * Return: 0 on success or a negative error code on failure.
+ */
+int xe_sriov_pf_control_prepare_flr(struct xe_device *xe, unsigned int vfid)
+{
+	struct xe_gt *gt;
+	unsigned int id;
+	int result = 0;
+	int err;
+
+	for_each_gt(gt, xe, id) {
+		err = xe_gt_sriov_pf_control_prepare_flr(gt, vfid);
+		result = result ? -EUCLEAN : err;
+	}
+
+	return result;
+}
+
 /**
  * xe_sriov_pf_control_wait_flr() - Wait for a VF reset (FLR) to complete.
  * @xe: the &xe_device
diff --git a/drivers/gpu/drm/xe/xe_sriov_pf_control.h b/drivers/gpu/drm/xe/xe_sriov_pf_control.h
index ef9f219b2109..74981a67db88 100644
--- a/drivers/gpu/drm/xe/xe_sriov_pf_control.h
+++ b/drivers/gpu/drm/xe/xe_sriov_pf_control.h
@@ -12,6 +12,7 @@ int xe_sriov_pf_control_pause_vf(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_resume_vf(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_stop_vf(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_reset_vf(struct xe_device *xe, unsigned int vfid);
+int xe_sriov_pf_control_prepare_flr(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_wait_flr(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_sync_flr(struct xe_device *xe, unsigned int vfid);
 int xe_sriov_pf_control_trigger_save_vf(struct xe_device *xe, unsigned int vfid);
diff --git a/drivers/gpu/drm/xe/xe_sriov_vfio.c b/drivers/gpu/drm/xe/xe_sriov_vfio.c
index 3da81af97b8b..00f96b0976d1 100644
--- a/drivers/gpu/drm/xe/xe_sriov_vfio.c
+++ b/drivers/gpu/drm/xe/xe_sriov_vfio.c
@@ -42,6 +42,7 @@ _type xe_sriov_vfio_##_func(struct xe_device *xe, unsigned int vfid)		\
 EXPORT_SYMBOL_FOR_MODULES(xe_sriov_vfio_##_func, "xe-vfio-pci")
 
 DEFINE_XE_SRIOV_VFIO_FUNCTION(int, wait_flr_done, control_wait_flr);
+DEFINE_XE_SRIOV_VFIO_FUNCTION(int, flr_prepare, control_prepare_flr);
 DEFINE_XE_SRIOV_VFIO_FUNCTION(int, suspend_device, control_pause_vf);
 DEFINE_XE_SRIOV_VFIO_FUNCTION(int, resume_device, control_resume_vf);
 DEFINE_XE_SRIOV_VFIO_FUNCTION(int, stop_copy_enter, control_trigger_save_vf);
diff --git a/include/drm/intel/xe_sriov_vfio.h b/include/drm/intel/xe_sriov_vfio.h
index e9814e8149fd..27c224a70e6f 100644
--- a/include/drm/intel/xe_sriov_vfio.h
+++ b/include/drm/intel/xe_sriov_vfio.h
@@ -27,6 +27,17 @@ struct xe_device *xe_sriov_vfio_get_pf(struct pci_dev *pdev);
  */
 bool xe_sriov_vfio_migration_supported(struct xe_device *xe);
 
+/**
+ * xe_sriov_vfio_flr_prepare() - Notify PF that VF FLR prepare has started.
+ * @xe: the PF &xe_device obtained by calling xe_sriov_vfio_get_pf()
+ * @vfid: the VF identifier (can't be 0)
+ *
+ * This function marks VF FLR as pending before PF receives GuC FLR event.
+ *
+ * Return: 0 on success or a negative error code on failure.
+ */
+int xe_sriov_vfio_flr_prepare(struct xe_device *xe, unsigned int vfid);
+
 /**
  * xe_sriov_vfio_wait_flr_done() - Wait for VF FLR completion.
  * @xe: the PF &xe_device obtained by calling xe_sriov_vfio_get_pf()
-- 
2.34.1

[PATCH v2 2/2] vfio/xe: Notify PF about VF FLR in reset_prepare

[Piórkowski, Piotr]

From: Piotr Piórkowski <piotr.piorkowski@intel.com>

Hook into the PCI error handler reset_prepare() callback to notify
the PF about an upcoming VF FLR before reset_done() is executed.
This enables early FLR_PREPARE signaling and ensures that the PF is
aware of the reset before the completion wait begins.

Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
Cc: Michał Winiarski <michal.winiarski@intel.com>
---
 drivers/vfio/pci/xe/main.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c
index fff95b2d5dde..88acfcf840fc 100644
--- a/drivers/vfio/pci/xe/main.c
+++ b/drivers/vfio/pci/xe/main.c
@@ -85,6 +85,19 @@ static void xe_vfio_pci_state_mutex_unlock(struct xe_vfio_pci_core_device *xe_vd
 	spin_unlock(&xe_vdev->reset_lock);
 }
 
+static void xe_vfio_pci_reset_prepare(struct pci_dev *pdev)
+{
+	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
+	int ret;
+
+	if (!pdev->is_virtfn)
+		return;
+
+	ret = xe_sriov_vfio_flr_prepare(xe_vdev->xe, xe_vdev->vfid);
+	if (ret)
+		dev_err(&pdev->dev, "Failed to prepare FLR: %d\n", ret);
+}
+
 static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
 {
 	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
@@ -127,6 +140,7 @@ static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
 }
 
 static const struct pci_error_handlers xe_vfio_pci_err_handlers = {
+	.reset_prepare = xe_vfio_pci_reset_prepare,
 	.reset_done = xe_vfio_pci_reset_done,
 	.error_detected = vfio_pci_core_aer_err_detected,
 };
-- 
2.34.1

Re: [PATCH v2 2/2] vfio/xe: Notify PF about VF FLR in reset_prepare

[Alex Williamson]

On Mon, 9 Mar 2026 16:24:49 +0100
"Piórkowski, Piotr" <piotr.piorkowski@intel.com> wrote:

> From: Piotr Piórkowski <piotr.piorkowski@intel.com>
> 
> Hook into the PCI error handler reset_prepare() callback to notify
> the PF about an upcoming VF FLR before reset_done() is executed.
> This enables early FLR_PREPARE signaling and ensures that the PF is
> aware of the reset before the completion wait begins.
> 
> Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
> Cc: Michał Winiarski <michal.winiarski@intel.com>
> ---
>  drivers/vfio/pci/xe/main.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c
> index fff95b2d5dde..88acfcf840fc 100644
> --- a/drivers/vfio/pci/xe/main.c
> +++ b/drivers/vfio/pci/xe/main.c
> @@ -85,6 +85,19 @@ static void xe_vfio_pci_state_mutex_unlock(struct xe_vfio_pci_core_device *xe_vd
>  	spin_unlock(&xe_vdev->reset_lock);
>  }
>  
> +static void xe_vfio_pci_reset_prepare(struct pci_dev *pdev)
> +{
> +	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
> +	int ret;
> +
> +	if (!pdev->is_virtfn)
> +		return;
> +
> +	ret = xe_sriov_vfio_flr_prepare(xe_vdev->xe, xe_vdev->vfid);
> +	if (ret)
> +		dev_err(&pdev->dev, "Failed to prepare FLR: %d\n", ret);
> +}
> +
>  static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
>  {
>  	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
> @@ -127,6 +140,7 @@ static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
>  }
>  
>  static const struct pci_error_handlers xe_vfio_pci_err_handlers = {
> +	.reset_prepare = xe_vfio_pci_reset_prepare,
>  	.reset_done = xe_vfio_pci_reset_done,
>  	.error_detected = vfio_pci_core_aer_err_detected,
>  };

Looks ok to me, should really have Michał's ack as well.  Will this go
in through drm?

Reviewed-by: Alex Williamson <alex@shazbot.org>

Re: [PATCH v2 2/2] vfio/xe: Notify PF about VF FLR in reset_prepare

[Michał Winiarski]

On Tue, Mar 17, 2026 at 08:43:40AM -0600, Alex Williamson wrote:
> On Mon, 9 Mar 2026 16:24:49 +0100
> "Piórkowski, Piotr" <piotr.piorkowski@intel.com> wrote:
> 
> > From: Piotr Piórkowski <piotr.piorkowski@intel.com>
> > 
> > Hook into the PCI error handler reset_prepare() callback to notify
> > the PF about an upcoming VF FLR before reset_done() is executed.
> > This enables early FLR_PREPARE signaling and ensures that the PF is
> > aware of the reset before the completion wait begins.
> > 
> > Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
> > Cc: Michał Winiarski <michal.winiarski@intel.com>
> > ---
> >  drivers/vfio/pci/xe/main.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/drivers/vfio/pci/xe/main.c b/drivers/vfio/pci/xe/main.c
> > index fff95b2d5dde..88acfcf840fc 100644
> > --- a/drivers/vfio/pci/xe/main.c
> > +++ b/drivers/vfio/pci/xe/main.c
> > @@ -85,6 +85,19 @@ static void xe_vfio_pci_state_mutex_unlock(struct xe_vfio_pci_core_device *xe_vd
> >  	spin_unlock(&xe_vdev->reset_lock);
> >  }
> >  
> > +static void xe_vfio_pci_reset_prepare(struct pci_dev *pdev)
> > +{
> > +	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
> > +	int ret;
> > +
> > +	if (!pdev->is_virtfn)
> > +		return;
> > +
> > +	ret = xe_sriov_vfio_flr_prepare(xe_vdev->xe, xe_vdev->vfid);
> > +	if (ret)
> > +		dev_err(&pdev->dev, "Failed to prepare FLR: %d\n", ret);
> > +}
> > +
> >  static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
> >  {
> >  	struct xe_vfio_pci_core_device *xe_vdev = pci_get_drvdata(pdev);
> > @@ -127,6 +140,7 @@ static void xe_vfio_pci_reset_done(struct pci_dev *pdev)
> >  }
> >  
> >  static const struct pci_error_handlers xe_vfio_pci_err_handlers = {
> > +	.reset_prepare = xe_vfio_pci_reset_prepare,
> >  	.reset_done = xe_vfio_pci_reset_done,
> >  	.error_detected = vfio_pci_core_aer_err_detected,
> >  };
> 
> Looks ok to me, should really have Michał's ack as well.  Will this go
> in through drm?
> 
> Reviewed-by: Alex Williamson <alex@shazbot.org>

Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>

And yes, we can take it through DRM.

Thanks,
-Michał

Re: [PATCH v2 1/2] drm/xe/pf: Add FLR_PREPARE state to VF control flow

[Michał Winiarski]

On Mon, Mar 09, 2026 at 04:24:48PM +0100, Piórkowski, Piotr wrote:
> From: Piotr Piórkowski <piotr.piorkowski@intel.com>
> 
> Our xe-vfio-pci component relies on the confirmation from the PF
> that VF FLR processing has finished, but due to the notification
> latency on the HW/FW side, PF might be unaware yet of the already
> triggered VF FLR.
> 
> Update VF state machine with new FLR_PREPARE state that indicate
> imminent VF FLR notification and treat that as a begin of the FLR
> sequence. Also introduce function that xe-vfio-pci should call to
> guarantee correct synchronization.
> 
> v2: move PREPARE into WIP, update commit msg (Michal)
> 
> Signed-off-by: Piotr Piórkowski <piotr.piorkowski@intel.com>
> Co-developed-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>

Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>

Thanks,
-Michał

> ---
>  drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c   | 78 +++++++++++++++----
>  drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h   |  1 +
>  .../gpu/drm/xe/xe_gt_sriov_pf_control_types.h |  2 +
>  drivers/gpu/drm/xe/xe_sriov_pf_control.c      | 24 ++++++
>  drivers/gpu/drm/xe/xe_sriov_pf_control.h      |  1 +
>  drivers/gpu/drm/xe/xe_sriov_vfio.c            |  1 +
>  include/drm/intel/xe_sriov_vfio.h             | 11 +++
>  7 files changed, 102 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
> index 5cb705c7ee7a..058585f063a9 100644
> --- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
> +++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.c
> @@ -171,6 +171,7 @@ static const char *control_bit_to_string(enum xe_gt_sriov_control_bits bit)
>  	case XE_GT_SRIOV_STATE_##_X: return #_X
>  	CASE2STR(WIP);
>  	CASE2STR(FLR_WIP);
> +	CASE2STR(FLR_PREPARE);
>  	CASE2STR(FLR_SEND_START);
>  	CASE2STR(FLR_WAIT_GUC);
>  	CASE2STR(FLR_GUC_DONE);
> @@ -1486,11 +1487,15 @@ int xe_gt_sriov_pf_control_stop_vf(struct xe_gt *gt, unsigned int vfid)
>   * The VF FLR state machine looks like::
>   *
>   *	 (READY,PAUSED,STOPPED)<------------<--------------o
> - *	    |                                               \
> - *	   flr                                               \
> - *	    |                                                 \
> - *	....V..........................FLR_WIP...........      \
> - *	:    \                                          :       \
> + *	    |             |                                 \
> + *	   flr           prepare                             \
> + *	    |             |                                   \
> + *	....V.............V............FLR_WIP...........      \
> + *	:   |             |                             :       \
> + *	:   |    FLR_PREPARE                            :        |
> + *	:   |    /                                      :        |
> + *	:   \   flr                                     :        |
> + *	:    \ /                                        :        |
>   *	:     \   o----<----busy                        :        |
>   *	:      \ /            /                         :        |
>   *	:       FLR_SEND_START---failed----->-----------o--->(FLR_FAILED)<---o
> @@ -1539,20 +1544,28 @@ static void pf_enter_vf_flr_send_start(struct xe_gt *gt, unsigned int vfid)
>  	pf_queue_vf(gt, vfid);
>  }
>  
> -static void pf_enter_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
> +static bool pf_exit_vf_flr_prepare(struct xe_gt *gt, unsigned int vfid)
>  {
> -	if (!pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP)) {
> -		xe_gt_sriov_dbg(gt, "VF%u FLR is already in progress\n", vfid);
> -		return;
> -	}
> +	if (!pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE))
> +		return false;
>  
> -	pf_enter_vf_wip(gt, vfid);
>  	pf_enter_vf_flr_send_start(gt, vfid);
> +	return true;
> +}
> +
> +static bool pf_enter_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
> +{
> +	if (!pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP))
> +		return false;
> +
> +	pf_enter_vf_wip(gt, vfid);
> +	return true;
>  }
>  
>  static void pf_exit_vf_flr_wip(struct xe_gt *gt, unsigned int vfid)
>  {
>  	if (pf_exit_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_WIP)) {
> +		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE);
>  		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_SEND_FINISH);
>  		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_RESET_MMIO);
>  		pf_escape_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_RESET_DATA);
> @@ -1760,21 +1773,54 @@ static void pf_enter_vf_flr_guc_done(struct xe_gt *gt, unsigned int vfid)
>  }
>  
>  /**
> - * xe_gt_sriov_pf_control_trigger_flr - Start a VF FLR sequence.
> + * xe_gt_sriov_pf_control_prepare_flr() - Notify PF that VF FLR request was issued.
>   * @gt: the &xe_gt
>   * @vfid: the VF identifier
>   *
> + * This is an optional early notification path used to mark pending FLR before
> + * the GuC notifies the PF with a FLR event.
> + *
>   * This function is for PF only.
>   *
>   * Return: 0 on success or a negative error code on failure.
>   */
> -int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid)
> +int xe_gt_sriov_pf_control_prepare_flr(struct xe_gt *gt, unsigned int vfid)
>  {
> -	pf_enter_vf_flr_wip(gt, vfid);
> +	if (!pf_enter_vf_flr_wip(gt, vfid))
> +		return -EALREADY;
>  
> +	pf_enter_vf_state(gt, vfid, XE_GT_SRIOV_STATE_FLR_PREPARE);
>  	return 0;
>  }
>  
> +static int pf_begin_vf_flr(struct xe_gt *gt, unsigned int vfid)
> +{
> +	if (pf_enter_vf_flr_wip(gt, vfid)) {
> +		pf_enter_vf_flr_send_start(gt, vfid);
> +		return 0;
> +	}
> +
> +	if (pf_exit_vf_flr_prepare(gt, vfid))
> +		return 0;
> +
> +	xe_gt_sriov_dbg(gt, "VF%u FLR is already in progress\n", vfid);
> +	return -EALREADY;
> +}
> +
> +/**
> + * xe_gt_sriov_pf_control_trigger_flr - Start a VF FLR sequence.
> + * @gt: the &xe_gt
> + * @vfid: the VF identifier
> + *
> + * This function is for PF only.
> + *
> + * Return: 0 on success or a negative error code on failure.
> + */
> +int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid)
> +{
> +	return pf_begin_vf_flr(gt, vfid);
> +}
> +
>  /**
>   * xe_gt_sriov_pf_control_sync_flr() - Synchronize on the VF FLR checkpoint.
>   * @gt: the &xe_gt
> @@ -1879,9 +1925,9 @@ static void pf_handle_vf_flr(struct xe_gt *gt, u32 vfid)
>  
>  	if (needs_dispatch_flr(xe)) {
>  		for_each_gt(gtit, xe, gtid)
> -			pf_enter_vf_flr_wip(gtit, vfid);
> +			pf_begin_vf_flr(gtit, vfid);
>  	} else {
> -		pf_enter_vf_flr_wip(gt, vfid);
> +		pf_begin_vf_flr(gt, vfid);
>  	}
>  }
>  
> diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
> index c36c8767f3ad..23182a5c5fb8 100644
> --- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
> +++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control.h
> @@ -27,6 +27,7 @@ int xe_gt_sriov_pf_control_process_restore_data(struct xe_gt *gt, unsigned int v
>  int xe_gt_sriov_pf_control_trigger_restore_vf(struct xe_gt *gt, unsigned int vfid);
>  int xe_gt_sriov_pf_control_finish_restore_vf(struct xe_gt *gt, unsigned int vfid);
>  int xe_gt_sriov_pf_control_stop_vf(struct xe_gt *gt, unsigned int vfid);
> +int xe_gt_sriov_pf_control_prepare_flr(struct xe_gt *gt, unsigned int vfid);
>  int xe_gt_sriov_pf_control_trigger_flr(struct xe_gt *gt, unsigned int vfid);
>  int xe_gt_sriov_pf_control_sync_flr(struct xe_gt *gt, unsigned int vfid, bool sync);
>  int xe_gt_sriov_pf_control_wait_flr(struct xe_gt *gt, unsigned int vfid);
> diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
> index 6027ba05a7f2..e78c59e08adf 100644
> --- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
> +++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_control_types.h
> @@ -15,6 +15,7 @@
>   *
>   * @XE_GT_SRIOV_STATE_WIP: indicates that some operations are in progress.
>   * @XE_GT_SRIOV_STATE_FLR_WIP: indicates that a VF FLR is in progress.
> + * @XE_GT_SRIOV_STATE_FLR_PREPARE: indicates that the PF received early VF FLR prepare notification.
>   * @XE_GT_SRIOV_STATE_FLR_SEND_START: indicates that the PF wants to send a FLR START command.
>   * @XE_GT_SRIOV_STATE_FLR_WAIT_GUC: indicates that the PF awaits for a response from the GuC.
>   * @XE_GT_SRIOV_STATE_FLR_GUC_DONE: indicates that the PF has received a response from the GuC.
> @@ -56,6 +57,7 @@ enum xe_gt_sriov_control_bits {
>  	XE_GT_SRIOV_STATE_WIP = 1,
>  
>  	XE_GT_SRIOV_STATE_FLR_WIP,
> +	XE_GT_SRIOV_STATE_FLR_PREPARE,
>  	XE_GT_SRIOV_STATE_FLR_SEND_START,
>  	XE_GT_SRIOV_STATE_FLR_WAIT_GUC,
>  	XE_GT_SRIOV_STATE_FLR_GUC_DONE,
> diff --git a/drivers/gpu/drm/xe/xe_sriov_pf_control.c b/drivers/gpu/drm/xe/xe_sriov_pf_control.c
> index ed4b9820b06e..15b4341d7f12 100644
> --- a/drivers/gpu/drm/xe/xe_sriov_pf_control.c
> +++ b/drivers/gpu/drm/xe/xe_sriov_pf_control.c
> @@ -123,6 +123,30 @@ int xe_sriov_pf_control_reset_vf(struct xe_device *xe, unsigned int vfid)
>  	return result;
>  }
>  
> +/**
> + * xe_sriov_pf_control_prepare_flr() - Notify PF that VF FLR prepare has started.
> + * @xe: the &xe_device
> + * @vfid: the VF identifier
> + *
> + * This function is for PF only.
> + *
> + * Return: 0 on success or a negative error code on failure.
> + */
> +int xe_sriov_pf_control_prepare_flr(struct xe_device *xe, unsigned int vfid)
> +{
> +	struct xe_gt *gt;
> +	unsigned int id;
> +	int result = 0;
> +	int err;
> +
> +	for_each_gt(gt, xe, id) {
> +		err = xe_gt_sriov_pf_control_prepare_flr(gt, vfid);
> +		result = result ? -EUCLEAN : err;
> +	}
> +
> +	return result;
> +}
> +
>  /**
>   * xe_sriov_pf_control_wait_flr() - Wait for a VF reset (FLR) to complete.
>   * @xe: the &xe_device
> diff --git a/drivers/gpu/drm/xe/xe_sriov_pf_control.h b/drivers/gpu/drm/xe/xe_sriov_pf_control.h
> index ef9f219b2109..74981a67db88 100644
> --- a/drivers/gpu/drm/xe/xe_sriov_pf_control.h
> +++ b/drivers/gpu/drm/xe/xe_sriov_pf_control.h
> @@ -12,6 +12,7 @@ int xe_sriov_pf_control_pause_vf(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_resume_vf(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_stop_vf(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_reset_vf(struct xe_device *xe, unsigned int vfid);
> +int xe_sriov_pf_control_prepare_flr(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_wait_flr(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_sync_flr(struct xe_device *xe, unsigned int vfid);
>  int xe_sriov_pf_control_trigger_save_vf(struct xe_device *xe, unsigned int vfid);
> diff --git a/drivers/gpu/drm/xe/xe_sriov_vfio.c b/drivers/gpu/drm/xe/xe_sriov_vfio.c
> index 3da81af97b8b..00f96b0976d1 100644
> --- a/drivers/gpu/drm/xe/xe_sriov_vfio.c
> +++ b/drivers/gpu/drm/xe/xe_sriov_vfio.c
> @@ -42,6 +42,7 @@ _type xe_sriov_vfio_##_func(struct xe_device *xe, unsigned int vfid)		\
>  EXPORT_SYMBOL_FOR_MODULES(xe_sriov_vfio_##_func, "xe-vfio-pci")
>  
>  DEFINE_XE_SRIOV_VFIO_FUNCTION(int, wait_flr_done, control_wait_flr);
> +DEFINE_XE_SRIOV_VFIO_FUNCTION(int, flr_prepare, control_prepare_flr);
>  DEFINE_XE_SRIOV_VFIO_FUNCTION(int, suspend_device, control_pause_vf);
>  DEFINE_XE_SRIOV_VFIO_FUNCTION(int, resume_device, control_resume_vf);
>  DEFINE_XE_SRIOV_VFIO_FUNCTION(int, stop_copy_enter, control_trigger_save_vf);
> diff --git a/include/drm/intel/xe_sriov_vfio.h b/include/drm/intel/xe_sriov_vfio.h
> index e9814e8149fd..27c224a70e6f 100644
> --- a/include/drm/intel/xe_sriov_vfio.h
> +++ b/include/drm/intel/xe_sriov_vfio.h
> @@ -27,6 +27,17 @@ struct xe_device *xe_sriov_vfio_get_pf(struct pci_dev *pdev);
>   */
>  bool xe_sriov_vfio_migration_supported(struct xe_device *xe);
>  
> +/**
> + * xe_sriov_vfio_flr_prepare() - Notify PF that VF FLR prepare has started.
> + * @xe: the PF &xe_device obtained by calling xe_sriov_vfio_get_pf()
> + * @vfid: the VF identifier (can't be 0)
> + *
> + * This function marks VF FLR as pending before PF receives GuC FLR event.
> + *
> + * Return: 0 on success or a negative error code on failure.
> + */
> +int xe_sriov_vfio_flr_prepare(struct xe_device *xe, unsigned int vfid);
> +
>  /**
>   * xe_sriov_vfio_wait_flr_done() - Wait for VF FLR completion.
>   * @xe: the PF &xe_device obtained by calling xe_sriov_vfio_get_pf()
> -- 
> 2.34.1
>