[RFC 00/14] Add PXP HWDRM support

[RFC 00/14] Add PXP HWDRM support

[Daniele Ceraolo Spurio]

PXP (Protected Xe Path) allows execution and flip to display of protected
(i.e. encrypted) objects. The HW supports multiple types of PXP, but
this series only introduces support for PXP HWDRM, which is mainly
targeted at encrypting data that is going to be displayed.

Even though we only plan to support 1 type of PXP for now, the interface
has been designed to allow support for other PXP types to be added at a
later point in time.

A user is expected to mark both BO and exec_queues as using PXP and the
driver will make sure that PXP is running and that the encryption is
valid and that no execution happens with an outdated encryption.

RFC: The series is in a decent state and all my local tests are working
on MTL, but I'm seeing failures on LNL; those are currently trending to
being issues with the contents of the test batch, but I didn't want to
hold the series back until I was sure of that because there are a couple
of choices I made where I'd like feedback, so sending it as a RFC so I
can get that feedback while I complete the debug. The 2 areas in which
I'd like feedback are:

1) VM/BO changes:
    - PXP requires PPGTT submissions from inside the driver, which isn't
      currently supported. I've hooked it up (patch 3), but I'd like an
      expert eye to confirm I didn't do anything stupid.
    - We need to reject submissions using VMs with mapped invalid BOs.
      I've done this by checking all mapped BOs for PXP submissions
      (patch 11), but I'm open to ideas on how to do this differently.

2) API: as mentioned above, I've kept this flexible. Just wanted to
   confirm no one had anything against the approach. See patches 9-11. 

Cc: José Roberto de Souza <jose.souza@intel.com>
Cc: Alan Previn <alan.previn.teres.alexis@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Daniele Ceraolo Spurio (14):
  drm/xe: Make exec_queue_kill safe to call twice
  drm/xe/pxp: Initialize PXP structure and KCR reg
  drm/xe/pxp: Allocate PXP execution resources
  drm/xe/pxp: Add VCS inline termination support
  drm/xe/pxp: Add GSC session invalidation support
  drm/xe/pxp: Handle the PXP termination interrupt
  drm/xe/pxp: Add GSC session initialization support
  drm/xe/pxp: Add hooks to mark an exec queue as using PXP
  drm/xe/pxp: Add API to mark a queue as using PXP
  drm/xe/pxp: add a query for PXP status
  drm/xe/pxp: Add API to mark a BO as using PXP
  drm/xe/pxp: add PXP PM support
  drm/xe/pxp: Add PXP debugfs support
  drm/xe/pxp: Enable PXP for MTL

 drivers/gpu/drm/xe/Makefile                   |   3 +
 drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |  40 +
 .../xe/compat-i915-headers/pxp/intel_pxp.h    |  14 +-
 .../gpu/drm/xe/instructions/xe_instr_defs.h   |   1 +
 .../gpu/drm/xe/instructions/xe_mfx_commands.h |  29 +
 .../gpu/drm/xe/instructions/xe_mi_commands.h  |   5 +
 drivers/gpu/drm/xe/regs/xe_engine_regs.h      |   1 +
 drivers/gpu/drm/xe/regs/xe_gt_regs.h          |   8 +
 drivers/gpu/drm/xe/regs/xe_pxp_regs.h         |  23 +
 drivers/gpu/drm/xe/xe_bo.c                    | 100 ++-
 drivers/gpu/drm/xe/xe_bo.h                    |   5 +
 drivers/gpu/drm/xe/xe_bo_types.h              |   3 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   3 +
 drivers/gpu/drm/xe/xe_device.c                |   6 +
 drivers/gpu/drm/xe/xe_device_types.h          |   8 +-
 drivers/gpu/drm/xe/xe_exec.c                  |   6 +
 drivers/gpu/drm/xe/xe_exec_queue.c            |  48 +-
 drivers/gpu/drm/xe/xe_exec_queue.h            |   5 +
 drivers/gpu/drm/xe/xe_exec_queue_types.h      |   7 +
 drivers/gpu/drm/xe/xe_hw_engine.c             |   2 +-
 drivers/gpu/drm/xe/xe_irq.c                   |  20 +-
 drivers/gpu/drm/xe/xe_lrc.c                   |  14 +-
 drivers/gpu/drm/xe/xe_lrc.h                   |   7 +-
 drivers/gpu/drm/xe/xe_pci.c                   |   3 +
 drivers/gpu/drm/xe/xe_pm.c                    |  42 +-
 drivers/gpu/drm/xe/xe_pxp.c                   | 713 ++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp.h                   |  32 +
 drivers/gpu/drm/xe/xe_pxp_debugfs.c           | 120 +++
 drivers/gpu/drm/xe/xe_pxp_debugfs.h           |  13 +
 drivers/gpu/drm/xe/xe_pxp_submit.c            | 549 ++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.h            |  20 +
 drivers/gpu/drm/xe/xe_pxp_types.h             | 103 +++
 drivers/gpu/drm/xe/xe_query.c                 |  33 +
 drivers/gpu/drm/xe/xe_ring_ops.c              |   4 +-
 drivers/gpu/drm/xe/xe_vm.c                    | 148 +++-
 drivers/gpu/drm/xe/xe_vm.h                    |   8 +
 drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
 include/uapi/drm/xe_drm.h                     |  83 +-
 38 files changed, 2190 insertions(+), 40 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/instructions/xe_mfx_commands.h
 create mode 100644 drivers/gpu/drm/xe/regs/xe_pxp_regs.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_types.h

-- 
2.43.0

[RFC 01/14] drm/xe: Make exec_queue_kill safe to call twice

[Daniele Ceraolo Spurio]

An upcoming PXP patch will kill queues at runtime when a PXP
invalidation event occurs, so we need exec_queue_kill to be safe to call
multiple times.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 5b166fa03684..02f684c0330d 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -280,8 +280,10 @@ void xe_vm_remove_compute_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 		return;
 
 	down_write(&vm->lock);
-	list_del(&q->lr.link);
-	--vm->preempt.num_exec_queues;
+	if (!list_empty(&q->lr.link)) {
+		list_del_init(&q->lr.link);
+		--vm->preempt.num_exec_queues;
+	}
 	if (q->lr.pfence) {
 		dma_fence_enable_sw_signaling(q->lr.pfence);
 		dma_fence_put(q->lr.pfence);
-- 
2.43.0

[RFC 02/14] drm/xe/pxp: Initialize PXP structure and KCR reg

[Daniele Ceraolo Spurio]

As the first step towards adding PXP support, hook in the PXP init
function, allocate the PXP structure and initialize the KCR register to
allow PXP HWDRM sessions.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/Makefile                   |   1 +
 .../xe/compat-i915-headers/pxp/intel_pxp.h    |   4 +-
 drivers/gpu/drm/xe/regs/xe_pxp_regs.h         |  17 +++
 drivers/gpu/drm/xe/xe_device.c                |   6 +
 drivers/gpu/drm/xe/xe_device_types.h          |   8 +-
 drivers/gpu/drm/xe/xe_pci.c                   |   2 +
 drivers/gpu/drm/xe/xe_pxp.c                   | 103 ++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp.h                   |  16 +++
 drivers/gpu/drm/xe/xe_pxp_types.h             |  28 +++++
 9 files changed, 181 insertions(+), 4 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/regs/xe_pxp_regs.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp.h
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_types.h

diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
index b1e03bfe4a68..5f15e6dd5057 100644
--- a/drivers/gpu/drm/xe/Makefile
+++ b/drivers/gpu/drm/xe/Makefile
@@ -104,6 +104,7 @@ xe-y += xe_bb.o \
 	xe_preempt_fence.o \
 	xe_pt.o \
 	xe_pt_walk.o \
+	xe_pxp.o \
 	xe_query.o \
 	xe_range_fence.o \
 	xe_reg_sr.o \
diff --git a/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h b/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
index c2c30ece8f77..881680727452 100644
--- a/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
+++ b/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
@@ -10,9 +10,9 @@
 #include <linux/types.h>
 
 struct drm_i915_gem_object;
-struct intel_pxp;
+struct xe_pxp;
 
-static inline int intel_pxp_key_check(struct intel_pxp *pxp,
+static inline int intel_pxp_key_check(struct xe_pxp *pxp,
 				      struct drm_i915_gem_object *obj,
 				      bool assign)
 {
diff --git a/drivers/gpu/drm/xe/regs/xe_pxp_regs.h b/drivers/gpu/drm/xe/regs/xe_pxp_regs.h
new file mode 100644
index 000000000000..e6378f8e3295
--- /dev/null
+++ b/drivers/gpu/drm/xe/regs/xe_pxp_regs.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright(c) 2024, Intel Corporation. All rights reserved.
+ */
+
+#ifndef __XE_PXP_REGS_H__
+#define __XE_PXP_REGS_H__
+
+#include "regs/xe_regs.h"
+
+/* The following registers are only valid on platforms with a media GT */
+
+/* KCR enable/disable control */
+#define KCR_INIT				XE_REG(0x3860f0)
+#define   KCR_INIT_ALLOW_DISPLAY_ME_WRITES 	REG_BIT(14)
+
+#endif /* __XE_PXP_REGS_H__ */
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index cfda7cb5df2c..b7904f44c389 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -46,6 +46,7 @@
 #include "xe_pcode.h"
 #include "xe_perf.h"
 #include "xe_pm.h"
+#include "xe_pxp.h"
 #include "xe_query.h"
 #include "xe_sriov.h"
 #include "xe_tile.h"
@@ -677,6 +678,11 @@ int xe_device_probe(struct xe_device *xe)
 	if (err)
 		goto err_fini_oa;
 
+	/* A PXP init failure is not fatal */
+	err = xe_pxp_init(xe);
+	if (err && err != -ENOTSUPP)
+		drm_err(&xe->drm, "PXP initialization failed: %pe\n", ERR_PTR(err));
+
 	err = drm_dev_register(&xe->drm, 0);
 	if (err)
 		goto err_fini_display;
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index c37be471d11c..a31bd8892fcc 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -31,6 +31,7 @@
 
 struct xe_ggtt;
 struct xe_pat_ops;
+struct xe_pxp;
 
 #define XE_BO_INVALID_OFFSET	LONG_MAX
 
@@ -271,6 +272,8 @@ struct xe_device {
 		u8 has_llc:1;
 		/** @info.has_mmio_ext: Device has extra MMIO address range */
 		u8 has_mmio_ext:1;
+		/** @info.has_pxp: Device has PXP support */
+		u8 has_pxp:1;
 		/** @info.has_range_tlb_invalidation: Has range based TLB invalidations */
 		u8 has_range_tlb_invalidation:1;
 		/** @info.has_sriov: Supports SR-IOV */
@@ -466,6 +469,9 @@ struct xe_device {
 	/** @oa: oa perf counter subsystem */
 	struct xe_oa oa;
 
+	/** @pxp: Encapsulate Protected Xe Path support */
+	struct xe_pxp *pxp;
+
 	/** @needs_flr_on_fini: requests function-reset on fini */
 	bool needs_flr_on_fini;
 
@@ -530,8 +536,6 @@ struct xe_device {
 		unsigned int czclk_freq;
 		unsigned int fsb_freq, mem_freq, is_ddr3;
 	};
-
-	void *pxp;
 #endif
 };
 
diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c
index 53b67f39c7d5..de0d9f8bba2c 100644
--- a/drivers/gpu/drm/xe/xe_pci.c
+++ b/drivers/gpu/drm/xe/xe_pci.c
@@ -61,6 +61,7 @@ struct xe_device_desc {
 	u8 has_heci_gscfi:1;
 	u8 has_llc:1;
 	u8 has_mmio_ext:1;
+	u8 has_pxp:1;
 	u8 has_sriov:1;
 	u8 skip_guc_pc:1;
 	u8 skip_mtcfg:1;
@@ -609,6 +610,7 @@ static int xe_info_init_early(struct xe_device *xe,
 	xe->info.has_heci_gscfi = desc->has_heci_gscfi;
 	xe->info.has_llc = desc->has_llc;
 	xe->info.has_mmio_ext = desc->has_mmio_ext;
+	xe->info.has_pxp = desc->has_pxp;
 	xe->info.has_sriov = desc->has_sriov;
 	xe->info.skip_guc_pc = desc->skip_guc_pc;
 	xe->info.skip_mtcfg = desc->skip_mtcfg;
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
new file mode 100644
index 000000000000..cdb29b104006
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -0,0 +1,103 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright(c) 2024 Intel Corporation.
+ */
+
+#include "xe_pxp.h"
+
+#include <drm/drm_managed.h>
+
+#include "xe_device_types.h"
+#include "xe_force_wake.h"
+#include "xe_gt.h"
+#include "xe_gt_types.h"
+#include "xe_mmio.h"
+#include "xe_pxp_types.h"
+#include "xe_uc_fw.h"
+#include "regs/xe_pxp_regs.h"
+
+/**
+ * DOC: PXP
+ *
+ * PXP (Protected Xe Path) allows execution and flip to display of protected
+ * (i.e. encrypted) objects. This feature is currently only supported in
+ * integrated parts.
+ */
+
+static bool pxp_is_supported(const struct xe_device *xe)
+{
+	return xe->info.has_pxp;
+}
+
+static int kcr_pxp_set_status(const struct xe_pxp *pxp, bool enable)
+{
+	u32 val = enable ? _MASKED_BIT_ENABLE(KCR_INIT_ALLOW_DISPLAY_ME_WRITES) :
+		  _MASKED_BIT_DISABLE(KCR_INIT_ALLOW_DISPLAY_ME_WRITES);
+	int err;
+
+	err = xe_force_wake_get(gt_to_fw(pxp->gt), XE_FW_GT);
+	if (err)
+		return err;
+
+	xe_mmio_write32(pxp->gt, KCR_INIT, val);
+	XE_WARN_ON(xe_force_wake_put(gt_to_fw(pxp->gt), XE_FW_GT));
+
+	return 0;
+}
+
+static int kcr_pxp_enable(const struct xe_pxp *pxp)
+{
+	return kcr_pxp_set_status(pxp, true);
+}
+
+/**
+ * xe_pxp_init - initialize PXP support
+ * @xe: the xe_device structure
+ *
+ * Initialize the HW state and allocate the objects required for PXP support.
+ * Note that some of the requirement for PXP support (GSC proxy init, HuC auth)
+ * are performed asynchronously as part of the GSC init. PXP can only be used
+ * after both this function and the async worker have completed.
+ *
+ * Returns -ENOTSUPP if PXP is not supported, 0 if PXP initialization is
+ * successful, other errno value if there is an error during the init.
+ */
+int xe_pxp_init(struct xe_device *xe)
+{
+	struct xe_gt *gt = xe->tiles[0].media_gt;
+	struct xe_pxp *pxp;
+	int err;
+
+	if (!pxp_is_supported(xe))
+		return -ENOTSUPP;
+
+	/* we only support PXP on single tile devices with a media GT */
+	if (xe->info.tile_count > 1 || !gt)
+		return -ENOTSUPP;
+
+	/* The GSCCS is required for submissions to the GSC FW */
+	if (!(gt->info.engine_mask & BIT(XE_HW_ENGINE_GSCCS0)))
+		return -ENOTSUPP;
+
+	/* PXP requires both GSC and HuC firmwares to be available */
+	if (!xe_uc_fw_is_loadable(&gt->uc.gsc.fw) ||
+	    !xe_uc_fw_is_loadable(&gt->uc.huc.fw)) {
+		drm_info(&xe->drm, "skipping PXP init due to missing FW dependencies");
+		return -ENOTSUPP;
+	}
+
+	pxp = drmm_kzalloc(&xe->drm, sizeof(struct xe_pxp), GFP_KERNEL);
+	if (!pxp)
+		return -ENOMEM;
+
+	pxp->xe = xe;
+	pxp->gt = gt;
+
+	err = kcr_pxp_enable(pxp);
+	if (err)
+		return err;
+
+	xe->pxp = pxp;
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
new file mode 100644
index 000000000000..c47d31b9bdc9
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright(c) 2020, Intel Corporation. All rights reserved.
+ */
+
+#ifndef __XE_PXP_H__
+#define __XE_PXP_H__
+
+#include <linux/types.h>
+
+struct xe_device;
+struct xe_pxp;
+
+int xe_pxp_init(struct xe_device *xe);
+
+#endif /* __XE_PXP_H__ */
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
new file mode 100644
index 000000000000..1561e3bd2676
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright(c) 2020, Intel Corporation. All rights reserved.
+ */
+
+#ifndef __XE_PXP_TYPES_H__
+#define __XE_PXP_TYPES_H__
+
+#include <linux/types.h>
+
+struct xe_device;
+struct xe_gt;
+
+/**
+ * struct xe_pxp - pxp state
+ */
+struct xe_pxp {
+	/** @xe: Backpoiner to the xe_device struct */
+	struct xe_device *xe;
+
+	/**
+	 * @gt: pointer to the gt that owns the submission-side of PXP
+	 * (VDBOX, KCR and GSC)
+	 */
+	struct xe_gt *gt;
+};
+
+#endif /* __XE_PXP_TYPES_H__ */
-- 
2.43.0

[RFC 03/14] drm/xe/pxp: Allocate PXP execution resources

[Daniele Ceraolo Spurio]

PXP requires submissions to the HW for the following operations

1) Key invalidation, done via the VCS engine
2) Communication with the GSC FW for session management, done via the
   GSCCS

For #1 we can allocate a simple kernel context, but #2 requires the
submissions to be done with PPGTT, which is not currently supported in Xe.
To add this support, the following changes have been included:

- a new type of kernel-owned VM (marked as GSC)
- a new function to map a BO into a VM from within the kernel

RFC note: I've tweaked some of the VM functions to return the fence
further up the stack, so I can wait on it from the PXP code. Not sure if
this is the best approach.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 drivers/gpu/drm/xe/Makefile                   |   1 +
 drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |   7 +
 drivers/gpu/drm/xe/xe_exec_queue.c            |   3 +
 drivers/gpu/drm/xe/xe_pxp.c                   |  25 ++-
 drivers/gpu/drm/xe/xe_pxp_submit.c            | 188 ++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.h            |  16 ++
 drivers/gpu/drm/xe/xe_pxp_types.h             |  33 +++
 drivers/gpu/drm/xe/xe_vm.c                    | 100 +++++++++-
 drivers/gpu/drm/xe/xe_vm.h                    |   6 +
 drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
 10 files changed, 372 insertions(+), 8 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h

diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
index 5f15e6dd5057..a4514265085b 100644
--- a/drivers/gpu/drm/xe/Makefile
+++ b/drivers/gpu/drm/xe/Makefile
@@ -105,6 +105,7 @@ xe-y += xe_bb.o \
 	xe_pt.o \
 	xe_pt_walk.o \
 	xe_pxp.o \
+	xe_pxp_submit.o \
 	xe_query.o \
 	xe_range_fence.o \
 	xe_reg_sr.o \
diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
index 57520809e48d..f3c4cf10ba20 100644
--- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
+++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
@@ -6,6 +6,7 @@
 #ifndef _ABI_GSC_PXP_COMMANDS_ABI_H
 #define _ABI_GSC_PXP_COMMANDS_ABI_H
 
+#include <linux/sizes.h>
 #include <linux/types.h>
 
 /* Heci client ID for PXP commands */
@@ -13,6 +14,12 @@
 
 #define PXP_APIVER(x, y) (((x) & 0xFFFF) << 16 | ((y) & 0xFFFF))
 
+/*
+ * A PXP sub-section in an HECI packet can be up to 64K big in each direction.
+ * This does not include the top-level GSC header.
+ */
+#define PXP_MAX_PACKET_SIZE SZ_64K
+
 /*
  * there are a lot of status codes for PXP, but we only define the cross-API
  * common ones that we actually can handle in the kernel driver. Other failure
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
index 0ba37835849b..bc6e867aba17 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.c
+++ b/drivers/gpu/drm/xe/xe_exec_queue.c
@@ -131,6 +131,9 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
 	struct xe_exec_queue *q;
 	int err;
 
+	/* VMs for GSCCS queues (and only those) must have the XE_VM_FLAG_GSC flag */
+	xe_assert(xe, !vm || (!!(vm->flags & XE_VM_FLAG_GSC) == !!(hwe->engine_id == XE_HW_ENGINE_GSCCS0)));
+
 	q = __xe_exec_queue_alloc(xe, vm, logical_mask, width, hwe, flags,
 				  extensions);
 	if (IS_ERR(q))
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index cdb29b104006..01386b9f0c50 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -12,6 +12,7 @@
 #include "xe_gt.h"
 #include "xe_gt_types.h"
 #include "xe_mmio.h"
+#include "xe_pxp_submit.h"
 #include "xe_pxp_types.h"
 #include "xe_uc_fw.h"
 #include "regs/xe_pxp_regs.h"
@@ -50,6 +51,20 @@ static int kcr_pxp_enable(const struct xe_pxp *pxp)
 	return kcr_pxp_set_status(pxp, true);
 }
 
+static int kcr_pxp_disable(const struct xe_pxp *pxp)
+{
+	return kcr_pxp_set_status(pxp, false);
+}
+
+static void pxp_fini(void *arg)
+{
+	struct xe_pxp *pxp = arg;
+
+	xe_pxp_destroy_execution_resources(pxp);
+
+	/* no need to explicitly disable KCR since we're going to do an FLR */
+}
+
 /**
  * xe_pxp_init - initialize PXP support
  * @xe: the xe_device structure
@@ -97,7 +112,15 @@ int xe_pxp_init(struct xe_device *xe)
 	if (err)
 		return err;
 
+	err = xe_pxp_allocate_execution_resources(pxp);
+	if (err)
+		goto kcr_disable;
+
 	xe->pxp = pxp;
 
-	return 0;
+	return devm_add_action_or_reset(xe->drm.dev, pxp_fini, pxp);
+
+kcr_disable:
+	kcr_pxp_disable(pxp);
+	return err;
 }
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
new file mode 100644
index 000000000000..4fc3c7c58101
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright(c) 2024 Intel Corporation.
+ */
+
+#include "xe_pxp_submit.h"
+
+#include <drm/xe_drm.h>
+
+#include "xe_device_types.h"
+#include "xe_bo.h"
+#include "xe_exec_queue.h"
+#include "xe_gsc_submit.h"
+#include "xe_gt.h"
+#include "xe_pxp_types.h"
+#include "xe_vm.h"
+#include "regs/xe_gt_regs.h"
+
+static int create_vcs_context(struct xe_pxp *pxp)
+{
+	struct xe_gt *gt = pxp->gt;
+	struct xe_hw_engine *hwe;
+	struct xe_exec_queue *q;
+
+	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_VIDEO_DECODE, 0, true);
+	if (!hwe)
+		return -ENODEV;
+
+	q = xe_exec_queue_create(pxp->xe, NULL, BIT(hwe->logical_instance), 1, hwe,
+				 EXEC_QUEUE_FLAG_KERNEL | EXEC_QUEUE_FLAG_PERMANENT, 0);
+	if (IS_ERR(q))
+		return PTR_ERR(q);
+
+	pxp->vcs_queue = q;
+
+	return 0;
+}
+
+static void destroy_vcs_context(struct xe_pxp *pxp)
+{
+	if (pxp->vcs_queue)
+		xe_exec_queue_put(pxp->vcs_queue);
+}
+
+/*
+ * We allocate a single object for the batch and the input and output BOs. PXP
+ * commands can require a lot of BO space (see PXP_MAX_PACKET_SIZE), but we
+ * currently only support a subset of commands that are small (< 20 dwords),
+ * so a single page is enough for now.
+ */
+#define PXP_BB_SIZE		XE_PAGE_SIZE
+#define PXP_INOUT_SIZE		XE_PAGE_SIZE
+#define PXP_BO_SIZE		(PXP_BB_SIZE + (2 * PXP_INOUT_SIZE))
+#define PXP_BB_OFFSET		0
+#define PXP_MSG_IN_OFFSET 	PXP_BB_SIZE
+#define PXP_MSG_OUT_OFFSET 	(PXP_MSG_IN_OFFSET + PXP_INOUT_SIZE)
+static int allocate_gsc_execution_resources(struct xe_pxp *pxp)
+{
+	struct xe_gt *gt = pxp->gt;
+	struct xe_tile *tile = gt_to_tile(gt);
+	struct xe_device *xe = pxp->xe;
+	struct xe_hw_engine *hwe;
+	struct xe_vm *vm;
+	struct xe_bo *bo;
+	struct xe_exec_queue *q;
+	struct dma_fence *fence;
+	long timeout;
+	int err = 0;
+
+	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_OTHER, OTHER_GSC_INSTANCE, false);
+
+	/* we shouldn't reach here if the GSC engine is not available */
+	xe_assert(xe, hwe);
+
+	/* PXP instructions must be issued from PPGTT */
+	vm = xe_vm_create(xe, XE_VM_FLAG_GSC);
+	if (IS_ERR(vm))
+		return PTR_ERR(vm);
+
+	/* We allocate a single object for the batch and the in/out memory */
+	xe_vm_lock(vm, false);
+	bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BO_SIZE, ttm_bo_type_kernel,
+				  XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_NEEDS_UC);
+	xe_vm_unlock(vm);
+	if (IS_ERR(bo)) {
+		err = PTR_ERR(bo);
+		goto vm_out;
+	}
+
+	fence = xe_vm_bind_bo(vm, bo, NULL, 0, XE_CACHE_WB);
+	if (IS_ERR(fence)) {
+		err = PTR_ERR(fence);
+		goto bo_out;
+	}
+
+	timeout = dma_fence_wait_timeout(fence, false, HZ);
+	dma_fence_put(fence);
+	if (timeout <= 0) {
+		err = timeout ?: -ETIME;
+		goto bo_out;
+	}
+
+	q = xe_exec_queue_create(xe, vm, BIT(hwe->logical_instance), 1, hwe,
+				 EXEC_QUEUE_FLAG_KERNEL |
+				 EXEC_QUEUE_FLAG_PERMANENT, 0);
+	if (IS_ERR(q)) {
+		err = PTR_ERR(q);
+		goto bo_out;
+	}
+
+	pxp->gsc_exec.vm = vm;
+	pxp->gsc_exec.bo = bo;
+	pxp->gsc_exec.batch = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_BB_OFFSET);
+	pxp->gsc_exec.msg_in = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_IN_OFFSET);
+	pxp->gsc_exec.msg_out = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_OUT_OFFSET);
+	pxp->gsc_exec.q = q;
+
+	/* initialize host-session-handle (for all Xe-to-gsc-firmware PXP cmds) */
+	pxp->gsc_exec.host_session_handle = xe_gsc_create_host_session_id();
+
+	return 0;
+
+bo_out:
+	xe_vm_lock(vm, false);
+	xe_bo_unpin(bo);
+	xe_vm_unlock(vm);
+
+	xe_bo_put(bo);
+vm_out:
+	xe_vm_close_and_put(vm);
+
+	return err;
+}
+
+static void destroy_gsc_execution_resources(struct xe_pxp *pxp)
+{
+	if (!pxp->gsc_exec.q)
+		return;
+
+	iosys_map_clear(&pxp->gsc_exec.msg_out);
+	iosys_map_clear(&pxp->gsc_exec.msg_in);
+	iosys_map_clear(&pxp->gsc_exec.batch);
+
+	xe_exec_queue_put(pxp->gsc_exec.q);
+
+	xe_vm_lock(pxp->gsc_exec.vm, false);
+	xe_bo_unpin(pxp->gsc_exec.bo);
+	xe_vm_unlock(pxp->gsc_exec.vm);
+	xe_bo_put(pxp->gsc_exec.bo);
+
+	xe_vm_close_and_put(pxp->gsc_exec.vm);
+}
+
+/**
+ * xe_pxp_allocate_execution_resources - Allocate PXP submission objects
+ * @pxp: the xe_pxp structure
+ *
+ * Allocates exec_queues objects for VCS and GSCCS submission. The GSCCS
+ * submissions are done via PPGTT, so this function allocates a VM for it and
+ * maps the object into it.
+ *
+ * Returns 0 if the allocation and mapping is successful, an errno value
+ * otherwise.
+ */
+int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp)
+{
+	int err;
+
+	err = create_vcs_context(pxp);
+	if (err)
+		return err;
+
+	err = allocate_gsc_execution_resources(pxp);
+	if (err)
+		goto destroy_vcs_context;
+
+	return 0;
+
+destroy_vcs_context:
+	destroy_vcs_context(pxp);
+	return err;
+}
+
+void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp)
+{
+	destroy_gsc_execution_resources(pxp);
+	destroy_vcs_context(pxp);
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
new file mode 100644
index 000000000000..1a971fadc081
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright(c) 2024, Intel Corporation. All rights reserved.
+ */
+
+#ifndef __XE_PXP_SUBMIT_H__
+#define __XE_PXP_SUBMIT_H__
+
+#include <linux/types.h>
+
+struct xe_pxp;
+
+int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
+void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
+
+#endif /* __XE_PXP_SUBMIT_H__ */
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
index 1561e3bd2676..c16813253b47 100644
--- a/drivers/gpu/drm/xe/xe_pxp_types.h
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -6,10 +6,14 @@
 #ifndef __XE_PXP_TYPES_H__
 #define __XE_PXP_TYPES_H__
 
+#include <linux/iosys-map.h>
 #include <linux/types.h>
 
+struct xe_bo;
+struct xe_exec_queue;
 struct xe_device;
 struct xe_gt;
+struct xe_vm;
 
 /**
  * struct xe_pxp - pxp state
@@ -23,6 +27,35 @@ struct xe_pxp {
 	 * (VDBOX, KCR and GSC)
 	 */
 	struct xe_gt *gt;
+
+	/** @vcs_queue: kernel-owned VCS exec queue used for PXP operations */
+	struct xe_exec_queue *vcs_queue;
+
+	/** @gsc_exec: kernel-owned objects for PXP submissions to the GSCCS */
+	struct {
+		/**
+		 * @gsc_exec.host_session_handle: handle used in communications
+		 * with the GSC firmware.
+		 */
+		u64 host_session_handle;
+		/** @gsc_exec.vm: VM used for PXP submissions to the GSCCS */
+		struct xe_vm *vm;
+		/** @gsc_exec.q: GSCCS exec queue for PXP submissions */
+		struct xe_exec_queue *q;
+
+		/**
+		 * @gsc_exec.bo: BO used for submissions to the GSCCS and GSC
+		 * FW. It includes space for the GSCCS batch and the
+		 * input/output buffers read/written by the FW
+		 */
+		struct xe_bo *bo;
+		/** @gsc_exec.batch: iosys_map to the batch memory within the BO */
+		struct iosys_map batch;
+		/** @gsc_exec.msg_in: iosys_map to the input memory within the BO */
+		struct iosys_map msg_in;
+		/** @gsc_exec.msg_out: iosys_map to the output memory within the BO */
+		struct iosys_map msg_out;
+	} gsc_exec;
 };
 
 #endif /* __XE_PXP_TYPES_H__ */
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 02f684c0330d..412ec9cb9650 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1315,6 +1315,15 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 	struct xe_tile *tile;
 	u8 id;
 
+	/*
+	 * All GSC VMs are owned by the kernel and can also only be used on
+	 * the GSCCS. We don't want a kernel-owned VM to put the device in
+	 * either fault or not fault mode, so we need to exclude the GSC VMs
+	 * from that count; this is only safe if we ensure that all GSC VMs are
+	 * non-faulting.
+	 */
+	xe_assert(xe, !((flags & XE_VM_FLAG_GSC) && (flags & XE_VM_FLAG_FAULT_MODE)));
+
 	vm = kzalloc(sizeof(*vm), GFP_KERNEL);
 	if (!vm)
 		return ERR_PTR(-ENOMEM);
@@ -1442,7 +1451,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
 	mutex_lock(&xe->usm.lock);
 	if (flags & XE_VM_FLAG_FAULT_MODE)
 		xe->usm.num_vm_in_fault_mode++;
-	else if (!(flags & XE_VM_FLAG_MIGRATION))
+	else if (!(flags & (XE_VM_FLAG_MIGRATION | XE_VM_FLAG_GSC)))
 		xe->usm.num_vm_in_non_fault_mode++;
 	mutex_unlock(&xe->usm.lock);
 
@@ -2867,11 +2876,10 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
 	for (i = 0; i < vops->num_syncs; i++)
 		xe_sync_entry_signal(vops->syncs + i, fence);
 	xe_exec_queue_last_fence_set(wait_exec_queue, vm, fence);
-	dma_fence_put(fence);
 }
 
-static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
-				     struct xe_vma_ops *vops)
+static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,
+						   struct xe_vma_ops *vops)
 {
 	struct drm_exec exec;
 	struct dma_fence *fence;
@@ -2889,7 +2897,6 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
 
 		fence = ops_execute(vm, vops);
 		if (IS_ERR(fence)) {
-			err = PTR_ERR(fence);
 			/* FIXME: Killing VM rather than proper error handling */
 			xe_vm_kill(vm, false);
 			goto unlock;
@@ -2900,7 +2907,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
 
 unlock:
 	drm_exec_fini(&exec);
-	return err;
+	return fence;
 }
 
 #define SUPPORTED_FLAGS	\
@@ -3114,6 +3121,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 	struct xe_sync_entry *syncs = NULL;
 	struct drm_xe_vm_bind_op *bind_ops;
 	struct xe_vma_ops vops;
+	struct dma_fence *fence;
 	int err;
 	int i;
 
@@ -3264,7 +3272,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		goto unwind_ops;
 	}
 
-	err = vm_bind_ioctl_ops_execute(vm, &vops);
+	fence = vm_bind_ioctl_ops_execute(vm, &vops);
+	if (IS_ERR(fence))
+		err = PTR_ERR(fence);
+	else
+		dma_fence_put(fence);
 
 unwind_ops:
 	if (err && err != -ENODATA)
@@ -3297,6 +3309,80 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 	return err;
 }
 
+/**
+ * xe_vm_bind_bo - bind a kernel BO to a VM
+ * @vm: VM to bind the BO to
+ * @bo: BO to bind
+ * @q: exec queue to use for the bind (optional)
+ * @addr: address at which to bind the BO
+ * @cache_lvl: PAT cache level to use
+ *
+ * Execute a VM bind map operation on a kernel-owned BO to bind it into a
+ * kernel-owned VM.
+ *
+ * Returns 0 if the ops execution is successful, an errno value otherwise.
+ * TODO: return a fence instead.
+ */
+struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
+				struct xe_exec_queue *q, u64 addr,
+				enum xe_cache_level cache_lvl)
+{
+	struct xe_vma_ops vops;
+	struct drm_gpuva_ops *ops = NULL;
+	struct dma_fence *fence;
+	int err;
+
+	xe_bo_get(bo);
+	xe_vm_get(vm);
+	if (q)
+		xe_exec_queue_get(q);
+
+	down_write(&vm->lock);
+
+	xe_vma_ops_init(&vops, vm, q, NULL, 0);
+
+	ops = vm_bind_ioctl_ops_create(vm, bo, 0, addr, bo->size,
+				       DRM_XE_VM_BIND_OP_MAP, 0,
+				       vm->xe->pat.idx[cache_lvl], 0);
+	if (IS_ERR(ops)) {
+		err = PTR_ERR(ops);
+		goto release_vm_lock;
+	}
+
+	err = vm_bind_ioctl_ops_parse(vm, q, ops, NULL, 0, &vops, true);
+	if (err)
+		goto release_vm_lock;
+
+	/* Nothing to do */
+	if (list_empty(&vops.list)) {
+		err = -ENODATA;
+		goto unwind_ops;
+	}
+
+	fence = vm_bind_ioctl_ops_execute(vm, &vops);
+	if (IS_ERR(fence))
+		err = PTR_ERR(fence);
+
+unwind_ops:
+	if (err && err != -ENODATA)
+		vm_bind_ioctl_ops_unwind(vm, &ops, 1);
+
+	drm_gpuva_ops_free(&vm->gpuvm, ops);
+
+release_vm_lock:
+	up_write(&vm->lock);
+
+	if (q)
+		xe_exec_queue_put(q);
+	xe_vm_put(vm);
+	xe_bo_put(bo);
+
+	if (err)
+		fence = ERR_PTR(err);
+
+	return fence;
+}
+
 /**
  * xe_vm_lock() - Lock the vm's dma_resv object
  * @vm: The struct xe_vm whose lock is to be locked
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index b481608b12f1..5e298ac90dfc 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -19,6 +19,8 @@ struct drm_file;
 struct ttm_buffer_object;
 struct ttm_validate_buffer;
 
+struct dma_fence;
+
 struct xe_exec_queue;
 struct xe_file;
 struct xe_sync_entry;
@@ -248,6 +250,10 @@ int xe_vm_lock_vma(struct drm_exec *exec, struct xe_vma *vma);
 int xe_vm_validate_rebind(struct xe_vm *vm, struct drm_exec *exec,
 			  unsigned int num_fences);
 
+struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
+				struct xe_exec_queue *q, u64 addr,
+				enum xe_cache_level cache_lvl);
+
 /**
  * xe_vm_resv() - Return's the vm's reservation object
  * @vm: The vm
diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
index ce1a63a5e3e7..60ce327d303c 100644
--- a/drivers/gpu/drm/xe/xe_vm_types.h
+++ b/drivers/gpu/drm/xe/xe_vm_types.h
@@ -152,6 +152,7 @@ struct xe_vm {
 #define XE_VM_FLAG_BANNED		BIT(5)
 #define XE_VM_FLAG_TILE_ID(flags)	FIELD_GET(GENMASK(7, 6), flags)
 #define XE_VM_FLAG_SET_TILE_ID(tile)	FIELD_PREP(GENMASK(7, 6), (tile)->id)
+#define XE_VM_FLAG_GSC			BIT(8)
 	unsigned long flags;
 
 	/** @composite_fence_ctx: context composite fence */
-- 
2.43.0

[RFC 04/14] drm/xe/pxp: Add VCS inline termination support

[Daniele Ceraolo Spurio]

The key termination is done with a specific submission to the VCS
engine.

Note that this patch is meant to be squashed with the follow-up patches
that implement the other pieces of the termination flow. It is separate
for now for ease of review.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 .../gpu/drm/xe/instructions/xe_instr_defs.h   |   1 +
 .../gpu/drm/xe/instructions/xe_mfx_commands.h |  29 +++++
 .../gpu/drm/xe/instructions/xe_mi_commands.h  |   5 +
 drivers/gpu/drm/xe/xe_lrc.h                   |   3 +-
 drivers/gpu/drm/xe/xe_pxp_submit.c            | 100 ++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.h            |   2 +
 drivers/gpu/drm/xe/xe_ring_ops.c              |   4 +-
 7 files changed, 141 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/instructions/xe_mfx_commands.h

diff --git a/drivers/gpu/drm/xe/instructions/xe_instr_defs.h b/drivers/gpu/drm/xe/instructions/xe_instr_defs.h
index fd2ce7ace510..e559969468c4 100644
--- a/drivers/gpu/drm/xe/instructions/xe_instr_defs.h
+++ b/drivers/gpu/drm/xe/instructions/xe_instr_defs.h
@@ -16,6 +16,7 @@
 #define XE_INSTR_CMD_TYPE		GENMASK(31, 29)
 #define   XE_INSTR_MI			REG_FIELD_PREP(XE_INSTR_CMD_TYPE, 0x0)
 #define   XE_INSTR_GSC			REG_FIELD_PREP(XE_INSTR_CMD_TYPE, 0x2)
+#define   XE_INSTR_VIDEOPIPE		REG_FIELD_PREP(XE_INSTR_CMD_TYPE, 0x3)
 #define   XE_INSTR_GFXPIPE		REG_FIELD_PREP(XE_INSTR_CMD_TYPE, 0x3)
 #define   XE_INSTR_GFX_STATE		REG_FIELD_PREP(XE_INSTR_CMD_TYPE, 0x4)
 
diff --git a/drivers/gpu/drm/xe/instructions/xe_mfx_commands.h b/drivers/gpu/drm/xe/instructions/xe_mfx_commands.h
new file mode 100644
index 000000000000..a95fc96e49e3
--- /dev/null
+++ b/drivers/gpu/drm/xe/instructions/xe_mfx_commands.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef _XE_MFX_COMMANDS_H_
+#define _XE_MFX_COMMANDS_H_
+
+#include "instructions/xe_instr_defs.h"
+
+#define MFX_CMD_SUBTYPE		REG_GENMASK(28, 27) /* A.K.A cmd pipe */
+#define MFX_CMD_OPCODE		REG_GENMASK(26, 24)
+#define MFX_CMD_SUB_OPCODE	REG_GENMASK(23, 16)
+#define MFX_FLAGS_AND_LEN	REG_GENMASK(15, 0)
+
+#define XE_MFX_INSTR(subtype, op, sub_op, flags) \
+       (XE_INSTR_VIDEOPIPE | \
+	REG_FIELD_PREP(MFX_CMD_SUBTYPE, subtype) | \
+	REG_FIELD_PREP(MFX_CMD_OPCODE, op) | \
+	REG_FIELD_PREP(MFX_CMD_SUB_OPCODE, sub_op) | \
+	REG_FIELD_PREP(MFX_FLAGS_AND_LEN, flags))
+
+#define MFX_WAIT				XE_MFX_INSTR(1, 0, 0, 0)
+#define MFX_WAIT_DW0_PXP_SYNC_CONTROL_FLAG	REG_BIT(9)
+#define MFX_WAIT_DW0_MFX_SYNC_CONTROL_FLAG	REG_BIT(8)
+
+#define CRYPTO_KEY_EXCHANGE			XE_MFX_INSTR(2, 6, 9, 0)
+
+#endif
diff --git a/drivers/gpu/drm/xe/instructions/xe_mi_commands.h b/drivers/gpu/drm/xe/instructions/xe_mi_commands.h
index 10ec2920d31b..167fb0f742de 100644
--- a/drivers/gpu/drm/xe/instructions/xe_mi_commands.h
+++ b/drivers/gpu/drm/xe/instructions/xe_mi_commands.h
@@ -48,6 +48,7 @@
 #define   MI_LRI_LEN(x)			(((x) & 0xff) + 1)
 
 #define MI_FLUSH_DW			__MI_INSTR(0x26)
+#define   MI_FLUSH_DW_PROTECTED_MEM_EN	REG_BIT(22)
 #define   MI_FLUSH_DW_STORE_INDEX	REG_BIT(21)
 #define   MI_INVALIDATE_TLB		REG_BIT(18)
 #define   MI_FLUSH_DW_CCS		REG_BIT(16)
@@ -66,4 +67,8 @@
 
 #define MI_BATCH_BUFFER_START		__MI_INSTR(0x31)
 
+#define MI_SET_APPID			__MI_INSTR(0x0e)
+#define MI_SET_APPID_SESSION_ID_MASK	REG_GENMASK(6, 0)
+#define MI_SET_APPID_SESSION_ID(x)	REG_FIELD_PREP(MI_SET_APPID_SESSION_ID_MASK, x)
+
 #endif
diff --git a/drivers/gpu/drm/xe/xe_lrc.h b/drivers/gpu/drm/xe/xe_lrc.h
index c24542e89318..d411c3fbcbc6 100644
--- a/drivers/gpu/drm/xe/xe_lrc.h
+++ b/drivers/gpu/drm/xe/xe_lrc.h
@@ -20,7 +20,8 @@ struct xe_lrc;
 struct xe_lrc_snapshot;
 struct xe_vm;
 
-#define LRC_PPHWSP_SCRATCH_ADDR (0x34 * 4)
+#define LRC_PPHWSP_FLUSH_INVAL_SCRATCH_ADDR (0x34 * 4)
+#define LRC_PPHWSP_PXP_INVAL_SCRATCH_ADDR (0x40 * 4)
 
 struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
 			     u32 ring_size);
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
index 4fc3c7c58101..88200315a882 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.c
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
@@ -6,14 +6,19 @@
 #include "xe_pxp_submit.h"
 
 #include <drm/xe_drm.h>
+#include <linux/delay.h>
 
 #include "xe_device_types.h"
+#include "xe_bb.h"
 #include "xe_bo.h"
 #include "xe_exec_queue.h"
 #include "xe_gsc_submit.h"
 #include "xe_gt.h"
+#include "xe_lrc.h"
 #include "xe_pxp_types.h"
 #include "xe_vm.h"
+#include "instructions/xe_mfx_commands.h"
+#include "instructions/xe_mi_commands.h"
 #include "regs/xe_gt_regs.h"
 
 static int create_vcs_context(struct xe_pxp *pxp)
@@ -186,3 +191,98 @@ void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp)
 	destroy_gsc_execution_resources(pxp);
 	destroy_vcs_context(pxp);
 }
+
+/* stall until prior PXP and MFX/HCP/HUC objects are cmopleted */
+#define MFX_WAIT_PXP (MFX_WAIT | \
+		      MFX_WAIT_DW0_PXP_SYNC_CONTROL_FLAG | \
+		      MFX_WAIT_DW0_MFX_SYNC_CONTROL_FLAG)
+
+static void pxp_emit_wait(struct xe_bb *bb)
+{
+	/* wait for cmds to go through */
+	bb->cs[bb->len++] = MFX_WAIT_PXP;
+	bb->cs[bb->len++] = 0;
+}
+
+static void pxp_emit_session_selection(struct xe_bb *bb, u32 idx)
+{
+	pxp_emit_wait(bb);
+
+	/* pxp off */
+	bb->cs[bb->len++] = MI_FLUSH_DW | MI_FLUSH_IMM_DW;
+	bb->cs[bb->len++] = 0;
+	bb->cs[bb->len++] = 0;
+	bb->cs[bb->len++] = 0;
+
+	/* select session */
+	bb->cs[bb->len++] = MI_SET_APPID | MI_SET_APPID_SESSION_ID(idx);
+
+	bb->cs[bb->len++] = MFX_WAIT_PXP;
+
+	/* pxp on */
+	bb->cs[bb->len++] = MI_FLUSH_DW | MI_FLUSH_DW_PROTECTED_MEM_EN |
+			    MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_DW_STORE_INDEX |
+			    MI_FLUSH_IMM_DW;
+	bb->cs[bb->len++] = LRC_PPHWSP_PXP_INVAL_SCRATCH_ADDR | MI_FLUSH_DW_USE_GTT;
+	bb->cs[bb->len++] = 0;
+	bb->cs[bb->len++] = 0;
+
+	pxp_emit_wait(bb);
+}
+
+static void pxp_emit_inline_termination(struct xe_bb *bb)
+{
+	/* session inline termination */
+	bb->cs[bb->len++] = CRYPTO_KEY_EXCHANGE;
+	bb->cs[bb->len++] = 0;
+}
+
+static void pxp_emit_session_termination(struct xe_bb *bb, u32 idx)
+{
+	pxp_emit_session_selection(bb, idx);
+	pxp_emit_inline_termination(bb);
+}
+
+/**
+ * xe_pxp_submit_session_termination - submits a PXP inline termination
+ * @pxp: the xe_pxp structure
+ * @id: the session to terminate
+ *
+ * Emit an inline termination via the VCS engine to terminate a session.
+ *
+ * Returns 0 if the submission is successful, an errno value otherwise.
+ */
+int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id)
+{
+	struct xe_gt *gt = pxp->gt;
+	struct xe_sched_job *job;
+	struct xe_bb *bb;
+	struct dma_fence *fence;
+	long timeout;
+
+	bb = xe_bb_new(gt, SZ_4K, false);
+	if (IS_ERR(bb))
+		return PTR_ERR(bb);
+
+	pxp_emit_session_termination(bb, id);
+	pxp_emit_wait(bb);
+
+	job = xe_bb_create_job(pxp->vcs_queue, bb);
+	if (IS_ERR(job)) {
+		xe_bb_free(bb, NULL);
+		return PTR_ERR(job);
+	}
+
+	xe_sched_job_arm(job);
+	fence = dma_fence_get(&job->drm.s_fence->finished);
+	xe_sched_job_push(job);
+
+	timeout = dma_fence_wait_timeout(fence, false, HZ);
+
+	dma_fence_put(fence);
+	xe_bb_free(bb, NULL);
+	if (timeout <= 0)
+		return -EAGAIN;
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
index 1a971fadc081..4ee8c0acfed9 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.h
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
@@ -13,4 +13,6 @@ struct xe_pxp;
 int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
 void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
 
+int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id);
+
 #endif /* __XE_PXP_SUBMIT_H__ */
diff --git a/drivers/gpu/drm/xe/xe_ring_ops.c b/drivers/gpu/drm/xe/xe_ring_ops.c
index 0be4f489d3e1..a4b5a0f68a32 100644
--- a/drivers/gpu/drm/xe/xe_ring_ops.c
+++ b/drivers/gpu/drm/xe/xe_ring_ops.c
@@ -118,7 +118,7 @@ static int emit_flush_invalidate(u32 flag, u32 *dw, int i)
 	dw[i++] |= MI_INVALIDATE_TLB | MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_IMM_DW |
 		MI_FLUSH_DW_STORE_INDEX;
 
-	dw[i++] = LRC_PPHWSP_SCRATCH_ADDR | MI_FLUSH_DW_USE_GTT;
+	dw[i++] = LRC_PPHWSP_FLUSH_INVAL_SCRATCH_ADDR | MI_FLUSH_DW_USE_GTT;
 	dw[i++] = 0;
 	dw[i++] = ~0U;
 
@@ -156,7 +156,7 @@ static int emit_pipe_invalidate(u32 mask_flags, bool invalidate_tlb, u32 *dw,
 
 	flags &= ~mask_flags;
 
-	return emit_pipe_control(dw, i, 0, flags, LRC_PPHWSP_SCRATCH_ADDR, 0);
+	return emit_pipe_control(dw, i, 0, flags, LRC_PPHWSP_FLUSH_INVAL_SCRATCH_ADDR, 0);
 }
 
 static int emit_store_imm_ppgtt_posted(u64 addr, u64 value,
-- 
2.43.0

[RFC 05/14] drm/xe/pxp: Add GSC session invalidation support

[Daniele Ceraolo Spurio]

After a session is terminated, we need to inform the GSC so that it can
clean up its side of the allocation. This is done by sending an
invalidation command with the session ID.

Note that this patch is meant to be squashed with the follow-up patches
that implement the other pieces of the termination flow. It is separate
for now for ease of review.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |  12 +
 drivers/gpu/drm/xe/xe_pxp_submit.c            | 212 ++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.h            |   1 +
 3 files changed, 225 insertions(+)

diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
index f3c4cf10ba20..dddabeb4046e 100644
--- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
+++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
@@ -49,6 +49,7 @@ struct pxp_cmd_header {
 	u32 buffer_len;
 } __packed;
 
+#define PXP42_CMDID_INVALIDATE_STREAM_KEY 0x00000007
 #define PXP43_CMDID_NEW_HUC_AUTH 0x0000003F /* MTL+ */
 
 /* PXP-Input-Packet: HUC Auth-only */
@@ -63,4 +64,15 @@ struct pxp43_huc_auth_out {
 	struct pxp_cmd_header header;
 } __packed;
 
+/* PXP-Input-Packet: Invalidate Stream Key */
+struct pxp42_inv_stream_key_in {
+	struct pxp_cmd_header header;
+	u32 rsvd[3];
+} __packed;
+
+/* PXP-Output-Packet: Invalidate Stream Key */
+struct pxp42_inv_stream_key_out {
+	struct pxp_cmd_header header;
+	u32 rsvd;
+} __packed;
 #endif
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
index 88200315a882..e54e51e75567 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.c
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
@@ -15,8 +15,13 @@
 #include "xe_gsc_submit.h"
 #include "xe_gt.h"
 #include "xe_lrc.h"
+#include "xe_map.h"
 #include "xe_pxp_types.h"
+#include "xe_sched_job.h"
 #include "xe_vm.h"
+#include "abi/gsc_command_header_abi.h"
+#include "abi/gsc_pxp_commands_abi.h"
+#include "instructions/xe_gsc_commands.h"
 #include "instructions/xe_mfx_commands.h"
 #include "instructions/xe_mi_commands.h"
 #include "regs/xe_gt_regs.h"
@@ -286,3 +291,210 @@ int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id)
 
 	return 0;
 }
+
+static bool
+is_fw_err_platform_config(u32 type)
+{
+	switch (type) {
+	case PXP_STATUS_ERROR_API_VERSION:
+	case PXP_STATUS_PLATFCONFIG_KF1_NOVERIF:
+	case PXP_STATUS_PLATFCONFIG_KF1_BAD:
+		return true;
+	default:
+		break;
+	}
+	return false;
+}
+
+static const char *
+fw_err_to_string(u32 type)
+{
+	switch (type) {
+	case PXP_STATUS_ERROR_API_VERSION:
+		return "ERR_API_VERSION";
+	case PXP_STATUS_NOT_READY:
+		return "ERR_NOT_READY";
+	case PXP_STATUS_PLATFCONFIG_KF1_NOVERIF:
+	case PXP_STATUS_PLATFCONFIG_KF1_BAD:
+		return "ERR_PLATFORM_CONFIG";
+	default:
+		break;
+	}
+	return NULL;
+}
+
+static int pxp_pkt_submit(struct xe_exec_queue *q, u64 batch_addr)
+{
+	struct xe_gt *gt = q->gt;
+	struct xe_device *xe = gt_to_xe(gt);
+	struct xe_sched_job *job;
+	struct dma_fence *fence;
+	long timeout;
+
+	xe_assert(xe, q->hwe->engine_id == XE_HW_ENGINE_GSCCS0);
+
+	job =  xe_sched_job_create(q, &batch_addr);
+	if (IS_ERR(job))
+		return PTR_ERR(job);
+
+	xe_sched_job_arm(job);
+	fence = dma_fence_get(&job->drm.s_fence->finished);
+	xe_sched_job_push(job);
+
+	timeout = dma_fence_wait_timeout(fence, false, HZ);
+	dma_fence_put(fence);
+	if (timeout < 0)
+		return timeout;
+	else if (!timeout)
+		return -ETIME;
+
+	return 0;
+}
+
+static void emit_pxp_heci_cmd(struct xe_device *xe, struct iosys_map *batch,
+			      u64 addr_in, u32 size_in, u64 addr_out, u32 size_out)
+{
+	u32 len = 0;
+
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, GSC_HECI_CMD_PKT);
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, lower_32_bits(addr_in));
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, upper_32_bits(addr_in));
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, size_in);
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, lower_32_bits(addr_out));
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, upper_32_bits(addr_out));
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, size_out);
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, 0);
+	xe_map_wr(xe, batch, len++ * sizeof(u32), u32, MI_BATCH_BUFFER_END);
+}
+
+#define GSC_PENDING_RETRY_MAXCOUNT 40
+#define GSC_PENDING_RETRY_PAUSE_MS 50
+static int gsccs_send_message(struct xe_pxp *pxp, void *msg_in, size_t msg_in_size,
+			      void *msg_out, size_t msg_out_size_max)
+{
+	struct xe_device *xe = pxp->xe;
+	const size_t max_msg_size = PXP_INOUT_SIZE - sizeof(struct intel_gsc_mtl_header);
+	u32 wr_offset = 0;
+	u32 rd_offset = 0;
+	u32 reply_size;
+	int ret = 0;
+	int retry = GSC_PENDING_RETRY_MAXCOUNT;
+
+	if (!pxp->gsc_exec.q)
+		 return -ENODEV;
+
+	if (msg_in_size > max_msg_size || msg_out_size_max > max_msg_size)
+		 return -ENOSPC;
+
+	wr_offset = xe_gsc_emit_header(xe, &pxp->gsc_exec.msg_in, 0,
+				       HECI_MEADDRESS_PXP,
+				       pxp->gsc_exec.host_session_handle,
+				       msg_in_size);
+
+	/* NOTE: zero size packets are used for session-cleanups */
+	if (msg_in && msg_in_size)
+		 xe_map_memcpy_to(xe, &pxp->gsc_exec.msg_in, wr_offset,
+				  msg_in, msg_in_size);
+
+	/* Make sure the reply header does not contain stale data */
+	xe_gsc_poison_header(xe, &pxp->gsc_exec.msg_out, 0);
+
+	emit_pxp_heci_cmd(xe, &pxp->gsc_exec.batch, PXP_MSG_IN_OFFSET,
+			  wr_offset + msg_in_size, PXP_MSG_OUT_OFFSET,
+			  msg_out_size_max + wr_offset);
+
+	xe_device_wmb(xe);
+
+	do {
+		 ret = pxp_pkt_submit(pxp->gsc_exec.q, PXP_BB_OFFSET);
+		 if (ret)
+			  break;
+
+		 if (xe_gsc_check_and_update_pending(xe, &pxp->gsc_exec.msg_in, 0,
+							  &pxp->gsc_exec.msg_out, 0)) {
+			  ret = -EAGAIN;
+			  msleep(GSC_PENDING_RETRY_PAUSE_MS);
+		 }
+	} while (--retry && ret == -EAGAIN);
+
+	if (ret) {
+		 drm_err(&xe->drm, "failed to submit GSC PXP message: %d\n", ret);
+		 return ret;
+	}
+
+	ret = xe_gsc_read_out_header(xe, &pxp->gsc_exec.msg_out, 0,
+				     sizeof(struct pxp_cmd_header), &rd_offset);
+	if (ret) {
+		 drm_err(&xe->drm, "invalid GSC reply for PXP (err=%d)\n", ret);
+		 return ret;
+	}
+
+	reply_size = xe_map_rd_field(xe, &pxp->gsc_exec.msg_out, rd_offset,
+				     struct pxp_cmd_header, buffer_len);
+	reply_size += sizeof(struct pxp_cmd_header);
+
+	if (reply_size > msg_out_size_max) {
+		 drm_warn(&xe->drm, "caller with insufficient PXP reply size %u (%ld)\n",
+			   reply_size, msg_out_size_max);
+		 reply_size = msg_out_size_max;
+	}
+
+	if (msg_out)
+		  xe_map_memcpy_from(xe, msg_out, &pxp->gsc_exec.msg_out,
+					rd_offset, reply_size);
+
+	xe_gsc_poison_header(xe, &pxp->gsc_exec.msg_in, 0);
+
+	return ret;
+}
+
+/**
+ * xe_pxp_submit_session_invalidation - submits a PXP GSC invalidation
+ * @pxp: the xe_pxp structure
+ * @id: the session to invalidate
+ *
+ * Submit a message to the GSC FW to notify it that a session has been
+ * terminated and is therefore invalid.
+ *
+ * Returns 0 if the submission is successful, an errno value otherwise.
+ */
+int xe_pxp_submit_session_invalidation(struct xe_pxp *pxp, u32 id)
+{
+	struct xe_device *xe = pxp->xe;
+	struct pxp42_inv_stream_key_in msg_in = {0};
+	struct pxp42_inv_stream_key_out msg_out = {0};
+	int ret = 0;
+
+	/*
+	 * Stream key invalidation reuses the same version 4.2 input/output
+	 * command format but firmware requires 4.3 API interaction
+	 */
+	msg_in.header.api_version = PXP_APIVER(4, 3);
+	msg_in.header.command_id = PXP42_CMDID_INVALIDATE_STREAM_KEY;
+	msg_in.header.buffer_len = sizeof(msg_in) - sizeof(msg_in.header);
+
+	msg_in.header.stream_id = FIELD_PREP(PXP_CMDHDR_EXTDATA_SESSION_VALID, 1);
+	msg_in.header.stream_id |= FIELD_PREP(PXP_CMDHDR_EXTDATA_APP_TYPE, 0);
+	msg_in.header.stream_id |= FIELD_PREP(PXP_CMDHDR_EXTDATA_SESSION_ID, id);
+
+	ret = gsccs_send_message(pxp, &msg_in, sizeof(msg_in), &msg_out, sizeof(msg_out));
+	if (ret) {
+		drm_err(&xe->drm, "Failed to inv-stream-key-%u, ret=[%d]\n",
+			id, ret);
+	} else if (msg_out.header.status != 0) {
+		if (is_fw_err_platform_config(msg_out.header.status)) {
+			drm_info_once(&xe->drm,
+				      "PXP inv-stream-key-%u failed due to BIOS/SOC :0x%08x:%s\n",
+				      id, msg_out.header.status,
+				      fw_err_to_string(msg_out.header.status));
+		} else {
+			drm_dbg(&xe->drm, "PXP inv-stream-key-%u failed 0x%08x:%s:\n",
+				id, msg_out.header.status,
+				fw_err_to_string(msg_out.header.status));
+			drm_dbg(&xe->drm, "     cmd-detail: ID=[0x%08x],API-Ver-[0x%08x]\n",
+				msg_in.header.command_id, msg_in.header.api_version);
+		}
+	}
+
+	return ret;
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
index 4ee8c0acfed9..7f26f1bd90dc 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.h
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
@@ -14,5 +14,6 @@ int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
 void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
 
 int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id);
+int xe_pxp_submit_session_invalidation(struct xe_pxp *pxp, u32 session_id);
 
 #endif /* __XE_PXP_SUBMIT_H__ */
-- 
2.43.0

[RFC 06/14] drm/xe/pxp: Handle the PXP termination interrupt

[Daniele Ceraolo Spurio]

When something happen to the session, the HW generates a termination
interrupt. In reply to this, the driver is required to submit an inline
session termination via the VCS, trigger the global termination and
notify the GSC FW that the session is now invalid.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/regs/xe_gt_regs.h  |   8 ++
 drivers/gpu/drm/xe/regs/xe_pxp_regs.h |   6 ++
 drivers/gpu/drm/xe/xe_irq.c           |  20 +++-
 drivers/gpu/drm/xe/xe_pxp.c           | 140 +++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_pxp.h           |   3 +
 drivers/gpu/drm/xe/xe_pxp_types.h     |  13 +++
 6 files changed, 186 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/regs/xe_gt_regs.h b/drivers/gpu/drm/xe/regs/xe_gt_regs.h
index d44564bad009..4064e7827b24 100644
--- a/drivers/gpu/drm/xe/regs/xe_gt_regs.h
+++ b/drivers/gpu/drm/xe/regs/xe_gt_regs.h
@@ -547,6 +547,7 @@
 #define   ENGINE1_MASK				REG_GENMASK(31, 16)
 #define   ENGINE0_MASK				REG_GENMASK(15, 0)
 #define GPM_WGBOXPERF_INTR_ENABLE		XE_REG(0x19003c, XE_REG_OPTION_VF)
+#define CRYPTO_RSVD_INTR_ENABLE			XE_REG(0x190040)
 #define GUNIT_GSC_INTR_ENABLE			XE_REG(0x190044, XE_REG_OPTION_VF)
 #define CCS_RSVD_INTR_ENABLE			XE_REG(0x190048, XE_REG_OPTION_VF)
 
@@ -557,6 +558,7 @@
 #define   INTR_ENGINE_INTR(x)			REG_FIELD_GET(GENMASK(15, 0), x)
 #define   OTHER_GUC_INSTANCE			0
 #define   OTHER_GSC_HECI2_INSTANCE		3
+#define   OTHER_KCR_INSTANCE			4
 #define   OTHER_GSC_INSTANCE			6
 
 #define IIR_REG_SELECTOR(x)			XE_REG(0x190070 + ((x) * 4), XE_REG_OPTION_VF)
@@ -568,6 +570,7 @@
 #define HECI2_RSVD_INTR_MASK			XE_REG(0x1900e4)
 #define GUC_SG_INTR_MASK			XE_REG(0x1900e8, XE_REG_OPTION_VF)
 #define GPM_WGBOXPERF_INTR_MASK			XE_REG(0x1900ec, XE_REG_OPTION_VF)
+#define CRYPTO_RSVD_INTR_MASK			XE_REG(0x1900f0)
 #define GUNIT_GSC_INTR_MASK			XE_REG(0x1900f4, XE_REG_OPTION_VF)
 #define CCS0_CCS1_INTR_MASK			XE_REG(0x190100)
 #define CCS2_CCS3_INTR_MASK			XE_REG(0x190104)
@@ -582,4 +585,9 @@
 #define   GT_CS_MASTER_ERROR_INTERRUPT		REG_BIT(3)
 #define   GT_RENDER_USER_INTERRUPT		REG_BIT(0)
 
+/* irqs for OTHER_KCR_INSTANCE */
+#define   KCR_PXP_STATE_TERMINATED_INTERRUPT 		REG_BIT(1)
+#define   KCR_APP_TERMINATED_PER_FW_REQ_INTERRUPT 	REG_BIT(2)
+#define   KCR_PXP_STATE_RESET_COMPLETE_INTERRUPT 	REG_BIT(3)
+
 #endif
diff --git a/drivers/gpu/drm/xe/regs/xe_pxp_regs.h b/drivers/gpu/drm/xe/regs/xe_pxp_regs.h
index e6378f8e3295..cf98d9008711 100644
--- a/drivers/gpu/drm/xe/regs/xe_pxp_regs.h
+++ b/drivers/gpu/drm/xe/regs/xe_pxp_regs.h
@@ -14,4 +14,10 @@
 #define KCR_INIT				XE_REG(0x3860f0)
 #define   KCR_INIT_ALLOW_DISPLAY_ME_WRITES 	REG_BIT(14)
 
+/* KCR hwdrm session in play status 0-31 */
+#define KCR_SIP					XE_REG(0x386260)
+
+/* PXP global terminate register for session termination */
+#define KCR_GLOBAL_TERMINATE			XE_REG(0x3860f8)
+
 #endif /* __XE_PXP_REGS_H__ */
diff --git a/drivers/gpu/drm/xe/xe_irq.c b/drivers/gpu/drm/xe/xe_irq.c
index 85733f993d09..62a5bc9e2eee 100644
--- a/drivers/gpu/drm/xe/xe_irq.c
+++ b/drivers/gpu/drm/xe/xe_irq.c
@@ -20,6 +20,7 @@
 #include "xe_hw_engine.h"
 #include "xe_memirq.h"
 #include "xe_mmio.h"
+#include "xe_pxp.h"
 #include "xe_sriov.h"
 
 /*
@@ -202,6 +203,15 @@ void xe_irq_enable_hwe(struct xe_gt *gt)
 		}
 		if (heci_mask)
 			xe_mmio_write32(gt, HECI2_RSVD_INTR_MASK, ~(heci_mask << 16));
+
+		if (xe_pxp_is_supported(xe)) {
+			u32 kcr_mask = KCR_PXP_STATE_TERMINATED_INTERRUPT |
+				       KCR_APP_TERMINATED_PER_FW_REQ_INTERRUPT |
+				       KCR_PXP_STATE_RESET_COMPLETE_INTERRUPT;
+
+			xe_mmio_write32(gt, CRYPTO_RSVD_INTR_ENABLE, kcr_mask << 16);
+			xe_mmio_write32(gt, CRYPTO_RSVD_INTR_MASK, ~(kcr_mask << 16));
+		}
 	}
 }
 
@@ -324,9 +334,15 @@ static void gt_irq_handler(struct xe_tile *tile,
 			}
 
 			if (class == XE_ENGINE_CLASS_OTHER) {
-				/* HECI GSCFI interrupts come from outside of GT */
+				/*
+				 * HECI GSCFI interrupts come from outside of GT.
+				 * KCR irqs come from inside GT but are handled
+				 * by the global PXP subsystem.
+				 */
 				if (HAS_HECI_GSCFI(xe) && instance == OTHER_GSC_INSTANCE)
 					xe_heci_gsc_irq_handler(xe, intr_vec);
+				else if (instance == OTHER_KCR_INSTANCE)
+					xe_pxp_irq_handler(xe, intr_vec);
 				else
 					gt_other_irq_handler(engine_gt, instance, intr_vec);
 			}
@@ -510,6 +526,8 @@ static void gt_irq_reset(struct xe_tile *tile)
 		xe_mmio_write32(mmio, GUNIT_GSC_INTR_ENABLE, 0);
 		xe_mmio_write32(mmio, GUNIT_GSC_INTR_MASK, ~0);
 		xe_mmio_write32(mmio, HECI2_RSVD_INTR_MASK, ~0);
+		xe_mmio_write32(mmio, CRYPTO_RSVD_INTR_ENABLE, 0);
+		xe_mmio_write32(mmio, CRYPTO_RSVD_INTR_MASK, ~0);
 	}
 
 	xe_mmio_write32(mmio, GPM_WGBOXPERF_INTR_ENABLE, 0);
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index 01386b9f0c50..5f41fdab205d 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -12,9 +12,11 @@
 #include "xe_gt.h"
 #include "xe_gt_types.h"
 #include "xe_mmio.h"
+#include "xe_pm.h"
 #include "xe_pxp_submit.h"
 #include "xe_pxp_types.h"
 #include "xe_uc_fw.h"
+#include "regs/xe_gt_regs.h"
 #include "regs/xe_pxp_regs.h"
 
 /**
@@ -25,11 +27,135 @@
  * integrated parts.
  */
 
-static bool pxp_is_supported(const struct xe_device *xe)
+#define ARB_SESSION 0xF /* TODO: move to UAPI */
+
+bool xe_pxp_is_supported(const struct xe_device *xe)
 {
 	return xe->info.has_pxp;
 }
 
+static bool pxp_is_enabled(const struct xe_pxp *pxp)
+{
+	return pxp;
+}
+
+
+static int pxp_wait_for_session_state(struct xe_pxp *pxp, u32 id, bool in_play)
+{
+	struct xe_gt *gt = pxp->gt;
+	u32 mask = BIT(id);
+	int ret;
+
+	ret = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
+	if (ret)
+		return ret;
+
+	ret = xe_mmio_wait32(gt, KCR_SIP, mask, in_play ? mask : 0,
+			     250, NULL, false);
+	xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
+
+	return ret;
+}
+
+static void pxp_terminate(struct xe_pxp *pxp)
+{
+	int ret = 0;
+	struct xe_device *xe = pxp->xe;
+	struct xe_gt *gt = pxp->gt;
+
+	drm_dbg(&xe->drm, "Terminating PXP\n");
+
+	/* terminate the hw session */
+	ret = xe_pxp_submit_session_termination(pxp, ARB_SESSION);
+	if (ret)
+		goto out;
+
+	ret = pxp_wait_for_session_state(pxp, ARB_SESSION, false);
+	if (ret)
+		goto out;
+
+	/* Trigger full HW cleanup */
+	XE_WARN_ON(xe_force_wake_get(gt_to_fw(gt), XE_FW_GT));
+	xe_mmio_write32(gt, KCR_GLOBAL_TERMINATE, 1);
+	xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
+
+	/* now we can tell the GSC to clean up its own state */
+	ret = xe_pxp_submit_session_invalidation(pxp, ARB_SESSION);
+
+out:
+	if (ret)
+		drm_err(&xe->drm, "PXP termination failed: %pe\n", ERR_PTR(ret));
+	return;
+}
+
+static void pxp_terminate_complete(struct xe_pxp *pxp)
+{
+	/* TODO mark the session as ready to start */
+}
+
+static void pxp_irq_work(struct work_struct *work)
+{
+	struct xe_pxp *pxp = container_of(work, typeof(*pxp), irq.work);
+	struct xe_device *xe = pxp->xe;
+	u32 events = 0;
+
+	spin_lock_irq(&xe->irq.lock);
+	events = pxp->irq.events;
+	pxp->irq.events = 0;
+	spin_unlock_irq(&xe->irq.lock);
+
+	if (!events)
+		return;
+
+	/*
+	 * If we're processing a termination irq while suspending then don't
+	 * bother, we're going to re-init everything on resume anyway.
+	 */
+	if ((events & PXP_TERMINATION_REQUEST) && !xe_pm_runtime_get_if_active(xe))
+		return;
+
+	if (events & PXP_TERMINATION_REQUEST) {
+		events &= ~PXP_TERMINATION_COMPLETE;
+		pxp_terminate(pxp);
+	}
+
+	if (events & PXP_TERMINATION_COMPLETE)
+		pxp_terminate_complete(pxp);
+
+	if (events & PXP_TERMINATION_REQUEST)
+		xe_pm_runtime_put(xe);
+}
+
+/**
+ * xe_pxp_irq_handler - Handles PXP interrupts.
+ * @pxp: pointer to pxp struct
+ * @iir: interrupt vector
+ */
+void xe_pxp_irq_handler(struct xe_device *xe, u16 iir)
+{
+	struct xe_pxp *pxp = xe->pxp;
+
+	if (!pxp_is_enabled(pxp)) {
+		drm_err(&xe->drm, "PXP irq 0x%x received with PXP disabled!\n", iir);
+		return;
+	}
+
+	lockdep_assert_held(&xe->irq.lock);
+
+	if (unlikely(!iir))
+		return;
+
+	if (iir & (KCR_PXP_STATE_TERMINATED_INTERRUPT |
+		   KCR_APP_TERMINATED_PER_FW_REQ_INTERRUPT))
+		pxp->irq.events |= PXP_TERMINATION_REQUEST;
+
+	if (iir & KCR_PXP_STATE_RESET_COMPLETE_INTERRUPT)
+		pxp->irq.events |= PXP_TERMINATION_COMPLETE;
+
+	if (pxp->irq.events)
+		queue_work(pxp->irq.wq, &pxp->irq.work);
+}
+
 static int kcr_pxp_set_status(const struct xe_pxp *pxp, bool enable)
 {
 	u32 val = enable ? _MASKED_BIT_ENABLE(KCR_INIT_ALLOW_DISPLAY_ME_WRITES) :
@@ -60,6 +186,7 @@ static void pxp_fini(void *arg)
 {
 	struct xe_pxp *pxp = arg;
 
+	destroy_workqueue(pxp->irq.wq);
 	xe_pxp_destroy_execution_resources(pxp);
 
 	/* no need to explicitly disable KCR since we're going to do an FLR */
@@ -83,7 +210,7 @@ int xe_pxp_init(struct xe_device *xe)
 	struct xe_pxp *pxp;
 	int err;
 
-	if (!pxp_is_supported(xe))
+	if (!xe_pxp_is_supported(xe))
 		return -ENOTSUPP;
 
 	/* we only support PXP on single tile devices with a media GT */
@@ -105,12 +232,17 @@ int xe_pxp_init(struct xe_device *xe)
 	if (!pxp)
 		return -ENOMEM;
 
+	INIT_WORK(&pxp->irq.work, pxp_irq_work);
 	pxp->xe = xe;
 	pxp->gt = gt;
 
+	pxp->irq.wq = alloc_ordered_workqueue("pxp-wq", 0);
+	if (!pxp->irq.wq)
+		return -ENOMEM;
+
 	err = kcr_pxp_enable(pxp);
 	if (err)
-		return err;
+		goto out_wq;
 
 	err = xe_pxp_allocate_execution_resources(pxp);
 	if (err)
@@ -122,5 +254,7 @@ int xe_pxp_init(struct xe_device *xe)
 
 kcr_disable:
 	kcr_pxp_disable(pxp);
+out_wq:
+	destroy_workqueue(pxp->irq.wq);
 	return err;
 }
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index c47d31b9bdc9..b69678304183 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -11,6 +11,9 @@
 struct xe_device;
 struct xe_pxp;
 
+bool xe_pxp_is_supported(const struct xe_device *xe);
+
 int xe_pxp_init(struct xe_device *xe);
+void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
 
 #endif /* __XE_PXP_H__ */
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
index c16813253b47..b34aeb292e79 100644
--- a/drivers/gpu/drm/xe/xe_pxp_types.h
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -8,6 +8,7 @@
 
 #include <linux/iosys-map.h>
 #include <linux/types.h>
+#include <linux/workqueue.h>
 
 struct xe_bo;
 struct xe_exec_queue;
@@ -56,6 +57,18 @@ struct xe_pxp {
 		/** @gsc_exec.msg_out: iosys_map to the output memory within the BO */
 		struct iosys_map msg_out;
 	} gsc_exec;
+
+	/** @irq: wrapper for the worker and queue used for PXP irq support */
+	struct {
+		/** @irq.work: worker that manages irq events. */
+		struct work_struct work;
+		/** @irq.wq: workqueue on which to queue the irq work. */
+		struct workqueue_struct *wq;
+		/** @irq.events: pending events, protected with xe->irq.lock. */
+		u32 events;
+#define PXP_TERMINATION_REQUEST  BIT(0)
+#define PXP_TERMINATION_COMPLETE BIT(1)
+	} irq;
 };
 
 #endif /* __XE_PXP_TYPES_H__ */
-- 
2.43.0

[RFC 07/14] drm/xe/pxp: Add GSC session initialization support

[Daniele Ceraolo Spurio]

A session is initialized (i.e. started) by sending a message to the GSC.

Note that this patch is meant to be squashed with the follow-up patches
that implement the other pieces of the session initialization and queue
setup flow. It is separate for now for ease of review.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h | 21 ++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.c            | 49 +++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_submit.h            |  1 +
 3 files changed, 71 insertions(+)

diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
index dddabeb4046e..e12670bfe26f 100644
--- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
+++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
@@ -50,6 +50,7 @@ struct pxp_cmd_header {
 } __packed;
 
 #define PXP42_CMDID_INVALIDATE_STREAM_KEY 0x00000007
+#define PXP43_CMDID_INIT_SESSION 0x00000036
 #define PXP43_CMDID_NEW_HUC_AUTH 0x0000003F /* MTL+ */
 
 /* PXP-Input-Packet: HUC Auth-only */
@@ -64,6 +65,26 @@ struct pxp43_huc_auth_out {
 	struct pxp_cmd_header header;
 } __packed;
 
+/* PXP-Input-Packet: Init PXP session */
+struct pxp43_create_arb_in {
+	struct pxp_cmd_header header;
+		/* header.stream_id fields for vesion 4.3 of Init PXP session: */
+		#define PXP43_INIT_SESSION_VALID BIT(0)
+		#define PXP43_INIT_SESSION_APPTYPE BIT(1)
+		#define PXP43_INIT_SESSION_APPID GENMASK(17, 2)
+	u32 protection_mode;
+		#define PXP43_INIT_SESSION_PROTECTION_ARB 0x2
+	u32 sub_session_id;
+	u32 init_flags;
+	u32 rsvd[12];
+} __packed;
+
+/* PXP-Input-Packet: Init PXP session */
+struct pxp43_create_arb_out {
+	struct pxp_cmd_header header;
+	u32 rsvd[8];
+} __packed;
+
 /* PXP-Input-Packet: Invalidate Stream Key */
 struct pxp42_inv_stream_key_in {
 	struct pxp_cmd_header header;
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
index e54e51e75567..9c76eeb5535c 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.c
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
@@ -26,6 +26,8 @@
 #include "instructions/xe_mi_commands.h"
 #include "regs/xe_gt_regs.h"
 
+#define ARB_SESSION 0xF /* TODO: move to UAPI */
+
 static int create_vcs_context(struct xe_pxp *pxp)
 {
 	struct xe_gt *gt = pxp->gt;
@@ -448,6 +450,53 @@ static int gsccs_send_message(struct xe_pxp *pxp, void *msg_in, size_t msg_in_si
 	return ret;
 }
 
+/**
+ * xe_pxp_submit_session_init - submits a PXP GSC session initialization
+ * @pxp: the xe_pxp structure
+ * @id: the session to initialize
+ *
+ * Submit a message to the GSC FW to initialize (i.e. start) a PXP session.
+ *
+ * Returns 0 if the submission is successful, an errno value otherwise.
+ */
+int xe_pxp_submit_session_init(struct xe_pxp *pxp, int id)
+{
+	struct xe_device *xe = pxp->xe;
+	struct pxp43_create_arb_in msg_in = {0};
+	struct pxp43_create_arb_out msg_out = {0};
+	int ret;
+
+	msg_in.header.api_version = PXP_APIVER(4, 3);
+	msg_in.header.command_id = PXP43_CMDID_INIT_SESSION;
+	msg_in.header.stream_id = (FIELD_PREP(PXP43_INIT_SESSION_APPID, id) |
+				   FIELD_PREP(PXP43_INIT_SESSION_VALID, 1) |
+				   FIELD_PREP(PXP43_INIT_SESSION_APPTYPE, 0));
+	msg_in.header.buffer_len = sizeof(msg_in) - sizeof(msg_in.header);
+
+	if (id == ARB_SESSION)
+		msg_in.protection_mode = PXP43_INIT_SESSION_PROTECTION_ARB;
+
+	ret = gsccs_send_message(pxp, &msg_in, sizeof(msg_in), &msg_out, sizeof(msg_out));
+	if (ret) {
+		drm_err(&xe->drm, "Failed to init session %d, ret=[%d]\n", id, ret);
+	} else if (msg_out.header.status != 0) {
+		if (is_fw_err_platform_config(msg_out.header.status)) {
+			drm_info_once(&xe->drm,
+				      "PXP init-session-%d failed due to BIOS/SOC:0x%08x:%s\n",
+				      id, msg_out.header.status,
+				      fw_err_to_string(msg_out.header.status));
+		} else {
+			drm_dbg(&xe->drm, "PXP init-session-%d failed 0x%08x:%st:\n",
+				id, msg_out.header.status,
+				fw_err_to_string(msg_out.header.status));
+			drm_dbg(&xe->drm, "     cmd-detail: ID=[0x%08x],API-Ver-[0x%08x]\n",
+				msg_in.header.command_id, msg_in.header.api_version);
+		}
+	}
+
+	return ret;
+}
+
 /**
  * xe_pxp_submit_session_invalidation - submits a PXP GSC invalidation
  * @pxp: the xe_pxp structure
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
index 7f26f1bd90dc..cbd2a39915d3 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.h
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
@@ -13,6 +13,7 @@ struct xe_pxp;
 int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
 void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
 
+int xe_pxp_submit_session_init(struct xe_pxp *pxp, int id);
 int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id);
 int xe_pxp_submit_session_invalidation(struct xe_pxp *pxp, u32 session_id);
 
-- 
2.43.0

[RFC 08/14] drm/xe/pxp: Add hooks to mark an exec queue as using PXP

[Daniele Ceraolo Spurio]

Instead of having a PXP session always running, we only start it if a
user tells us they want their queue to use PXP (API for that coming in
the next patch). Also, if a PXP invalidation occurs, we want to mark
all the PXP queues as dead (code for that also coming in a follow up
patch), so we need to keep the queues in a list.

Note that this patch is meant to be squashed with the follow-up patch
that implement the other pieces of the queue management flow. It is
separate for now for ease of review.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/xe_exec_queue_types.h |   7 +
 drivers/gpu/drm/xe/xe_pxp.c              | 234 ++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_pxp.h              |   4 +
 drivers/gpu/drm/xe/xe_pxp_types.h        |  19 ++
 4 files changed, 262 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_exec_queue_types.h b/drivers/gpu/drm/xe/xe_exec_queue_types.h
index 201588ec33c3..08ca3344e2be 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_exec_queue_types.h
@@ -127,6 +127,13 @@ struct xe_exec_queue {
 		spinlock_t lock;
 	} lr;
 
+	struct {
+		/** @pxp.type: PXP session type used by this queue */
+		u8 type;
+		/** @pxp.link: link into the list of PXP exec queues */
+		struct list_head link;
+	} pxp;
+
 	/** @ops: submission backend exec queue operations */
 	const struct xe_exec_queue_ops *ops;
 
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index 5f41fdab205d..159761e19e9b 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -8,9 +8,12 @@
 #include <drm/drm_managed.h>
 
 #include "xe_device_types.h"
+#include "xe_exec_queue_types.h"
 #include "xe_force_wake.h"
+#include "xe_gsc_proxy.h"
 #include "xe_gt.h"
 #include "xe_gt_types.h"
+#include "xe_huc.h"
 #include "xe_mmio.h"
 #include "xe_pm.h"
 #include "xe_pxp_submit.h"
@@ -39,6 +42,32 @@ static bool pxp_is_enabled(const struct xe_pxp *pxp)
 	return pxp;
 }
 
+static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
+{
+	bool ready;
+
+	XE_WARN_ON(xe_force_wake_get(gt_to_fw(pxp->gt), XE_FW_GSC));
+
+	/* PXP requires both HuC authentication via GSC and GSC proxy initialized */
+	ready = xe_huc_is_authenticated(&pxp->gt->uc.huc, XE_HUC_AUTH_VIA_GSC) &&
+		xe_gsc_proxy_init_done(&pxp->gt->uc.gsc);
+
+	xe_force_wake_put(gt_to_fw(pxp->gt), XE_FW_GSC);
+
+	return ready;
+}
+
+static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
+{
+	struct xe_gt *gt = pxp->gt;
+	u32 sip = 0;
+
+	XE_WARN_ON(xe_force_wake_get(gt_to_fw(gt), XE_FW_GT));
+	sip = xe_mmio_read32(gt, KCR_SIP);
+	xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
+
+	return sip & BIT(id);
+}
 
 static int pxp_wait_for_session_state(struct xe_pxp *pxp, u32 id, bool in_play)
 {
@@ -63,6 +92,20 @@ static void pxp_terminate(struct xe_pxp *pxp)
 	struct xe_device *xe = pxp->xe;
 	struct xe_gt *gt = pxp->gt;
 
+	/*
+	 * If we have a termination already in progress, we need to wait for
+	 * it to complete before queueing another one. We update the state
+	 * to signal that another termination is required and leave it to the
+	 * pxp_start() call to take care of it.
+	 */
+	if (!completion_done(&pxp->termination)) {
+		pxp->status = XE_PXP_NEEDS_TERMINATION;
+		return;
+	}
+
+	reinit_completion(&pxp->termination);
+	pxp->status = XE_PXP_TERMINATION_IN_PROGRESS;
+
 	drm_dbg(&xe->drm, "Terminating PXP\n");
 
 	/* terminate the hw session */
@@ -83,14 +126,34 @@ static void pxp_terminate(struct xe_pxp *pxp)
 	ret = xe_pxp_submit_session_invalidation(pxp, ARB_SESSION);
 
 out:
-	if (ret)
+	if (ret) {
 		drm_err(&xe->drm, "PXP termination failed: %pe\n", ERR_PTR(ret));
+		pxp->status = XE_PXP_ERROR;
+		complete_all(&pxp->termination);
+	}
+
 	return;
 }
 
 static void pxp_terminate_complete(struct xe_pxp *pxp)
 {
-	/* TODO mark the session as ready to start */
+	/*
+	 * We expect PXP to be in one of 2 states when we get here:
+	 * - XE_PXP_TERMINATION_IN_PROGRESS: a single termination event was
+	 * requested and it is now completing, so we're ready to start.
+	 * - XE_PXP_NEEDS_TERMINATION: a second termination was requested while
+	 * the first one was still being processed; we don't update the state
+	 * in this case so the pxp_start code will automatically issue that
+	 * second termination.
+	 */
+	if (pxp->status == XE_PXP_TERMINATION_IN_PROGRESS)
+		pxp->status = XE_PXP_READY_TO_START;
+	else if (pxp->status != XE_PXP_NEEDS_TERMINATION)
+		drm_err(&pxp->xe->drm,
+			"PXP termination complete while status was %u\n",
+			pxp->status);
+
+	complete_all(&pxp->termination);
 }
 
 static void pxp_irq_work(struct work_struct *work)
@@ -114,6 +177,8 @@ static void pxp_irq_work(struct work_struct *work)
 	if ((events & PXP_TERMINATION_REQUEST) && !xe_pm_runtime_get_if_active(xe))
 		return;
 
+	mutex_lock(&pxp->mutex);
+
 	if (events & PXP_TERMINATION_REQUEST) {
 		events &= ~PXP_TERMINATION_COMPLETE;
 		pxp_terminate(pxp);
@@ -122,6 +187,8 @@ static void pxp_irq_work(struct work_struct *work)
 	if (events & PXP_TERMINATION_COMPLETE)
 		pxp_terminate_complete(pxp);
 
+	mutex_unlock(&pxp->mutex);
+
 	if (events & PXP_TERMINATION_REQUEST)
 		xe_pm_runtime_put(xe);
 }
@@ -232,10 +299,21 @@ int xe_pxp_init(struct xe_device *xe)
 	if (!pxp)
 		return -ENOMEM;
 
+	INIT_LIST_HEAD(&pxp->queue_list);
 	INIT_WORK(&pxp->irq.work, pxp_irq_work);
 	pxp->xe = xe;
 	pxp->gt = gt;
 
+	/*
+	 * we'll use the completion to check if there is a termination pending,
+	 * so we start it as completed and we reinit it when a termination
+	 * is triggered.
+	 */
+	init_completion(&pxp->termination);
+	complete_all(&pxp->termination);
+
+	mutex_init(&pxp->mutex);
+
 	pxp->irq.wq = alloc_ordered_workqueue("pxp-wq", 0);
 	if (!pxp->irq.wq)
 		return -ENOMEM;
@@ -258,3 +336,155 @@ int xe_pxp_init(struct xe_device *xe)
 	destroy_workqueue(pxp->irq.wq);
 	return err;
 }
+
+static int __pxp_start_session(struct xe_pxp *pxp, u32 id)
+{
+	int ret;
+
+	if (pxp_session_is_in_play(pxp, id))
+		return -EEXIST;
+
+	ret = xe_pxp_submit_session_init(pxp, id);
+	if (ret) {
+		drm_err(&pxp->xe->drm, "Failed to init PXP session %u\n", id);
+		goto out;
+	}
+
+	ret = pxp_wait_for_session_state(pxp, id, true);
+	if (ret) {
+		drm_err(&pxp->xe->drm, "PXP session %u failed to go in play\n", id);
+		goto out;
+	}
+
+	drm_dbg(&pxp->xe->drm, "PXP session %u is active\n", id);
+
+out:
+	if (!ret)
+		pxp->status = XE_PXP_ACTIVE;
+	else
+		pxp->status = XE_PXP_ERROR;
+
+	return ret;
+}
+
+/**
+ * xe_pxp_exec_queue_add - add a queue from the PXP list
+ * @pxp: the xe->pxp pointer (it will be NULL if PXP is disabled)
+ * @q: the queue to add from the list
+ * @type: the type of PXP session this queue will use
+ *
+ * If PXP is enabled and the prerequisites are done, start the PXP default
+ * session (if not already running) and add the queue to the PXP list.
+ *
+ * Returns 0 if the PXP session is running and the queue is in the list,
+ * -ENODEV if PXP is disabled, -EBUSY if the PXP prerequisites are not done,
+ * other errno value if something goes wrong during the session start.
+ */
+#define PXP_TERMINATION_TIMEOUT_MS 500
+int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type)
+{
+	int ret = 0;
+
+	if (!pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	/*
+	 * Runtime suspend kills PXP, so we need to turn it off while we have
+	 * active queues that use PXP
+	 */
+	xe_pm_runtime_get(pxp->xe);
+
+	if (!pxp_prerequisites_done(pxp)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+wait_for_termination:
+	/*
+	 * if there is a termination in progress, wait for it.
+	 * We need to wait outside the lock because the completion is done from
+	 * within the lock
+	 */
+	if (!wait_for_completion_timeout(&pxp->termination,
+					 msecs_to_jiffies(PXP_TERMINATION_TIMEOUT_MS)))
+		return -ETIMEDOUT;
+
+	mutex_lock(&pxp->mutex);
+
+	/*
+	 * check if a new termination was issued between the above check and
+	 * grabbing the mutex
+	 */
+	if (!completion_done(&pxp->termination)) {
+		mutex_unlock(&pxp->mutex);
+		goto wait_for_termination;
+	}
+
+	/* If PXP is not already active, turn it on */
+	switch (pxp->status) {
+	case XE_PXP_ERROR:
+		ret = -EIO;
+		break;
+	case XE_PXP_ACTIVE:
+		break;
+	case XE_PXP_READY_TO_START:
+		ret = __pxp_start_session(pxp, ARB_SESSION);
+		break;
+	case XE_PXP_NEEDS_TERMINATION:
+		pxp_terminate(pxp);
+		mutex_unlock(&pxp->mutex);
+		goto wait_for_termination;
+	default:
+		drm_err(&pxp->xe->drm, "unexpected state during PXP start: %u", pxp->status);
+		ret = -EIO;
+		break;
+	}
+
+	/* If everything went ok, add the queue to the list */
+	if (!ret) {
+		list_add_tail(&q->pxp.link, &pxp->queue_list);
+		q->pxp.type = type;
+	}
+
+	mutex_unlock(&pxp->mutex);
+
+out:
+	/*
+	 * in the successful case the PM ref is released from
+	 * xe_pxp_exec_queue_remove
+	 */
+	if (ret)
+		xe_pm_runtime_put(pxp->xe);
+
+	return ret;
+}
+
+/**
+ * xe_pxp_exec_queue_remove - remove a queue from the PXP list
+ * @pxp: the xe->pxp pointer (it will be NULL if PXP is disabled)
+ * @q: the queue to remove from the list
+ *
+ * If PXP is enabled and the exec_queue is in the list, the queue will be
+ * removed from the list and its PM reference will be released. It is safe to
+ * call this function multiple times for the same queue.
+ */
+void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q)
+{
+	bool need_pm_put = false;
+
+	if (!pxp_is_enabled(pxp))
+		return;
+
+	mutex_lock(&pxp->mutex);
+
+	if (!list_empty(&q->pxp.link)) {
+		list_del_init(&q->pxp.link);
+		q->pxp.type = 0;
+		need_pm_put = true;
+	}
+
+	mutex_unlock(&pxp->mutex);
+
+	if (need_pm_put)
+		xe_pm_runtime_put(pxp->xe);
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index b69678304183..6c99af072ab0 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -9,6 +9,7 @@
 #include <linux/types.h>
 
 struct xe_device;
+struct xe_exec_queue;
 struct xe_pxp;
 
 bool xe_pxp_is_supported(const struct xe_device *xe);
@@ -16,4 +17,7 @@ bool xe_pxp_is_supported(const struct xe_device *xe);
 int xe_pxp_init(struct xe_device *xe);
 void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
 
+int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type);
+void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q);
+
 #endif /* __XE_PXP_H__ */
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
index b34aeb292e79..5f7b031cafb1 100644
--- a/drivers/gpu/drm/xe/xe_pxp_types.h
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -6,7 +6,9 @@
 #ifndef __XE_PXP_TYPES_H__
 #define __XE_PXP_TYPES_H__
 
+#include <linux/completion.h>
 #include <linux/iosys-map.h>
+#include <linux/mutex.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 
@@ -16,6 +18,14 @@ struct xe_device;
 struct xe_gt;
 struct xe_vm;
 
+enum xe_pxp_status {
+	XE_PXP_ERROR = -1,
+	XE_PXP_NEEDS_TERMINATION = 0, /* starting status */
+	XE_PXP_TERMINATION_IN_PROGRESS,
+	XE_PXP_READY_TO_START,
+	XE_PXP_ACTIVE
+};
+
 /**
  * struct xe_pxp - pxp state
  */
@@ -69,6 +79,15 @@ struct xe_pxp {
 #define PXP_TERMINATION_REQUEST  BIT(0)
 #define PXP_TERMINATION_COMPLETE BIT(1)
 	} irq;
+
+	/** @mutex: protects the pxp status and the queue list */
+	struct mutex mutex;
+	/** @status: the current pxp status */
+	enum xe_pxp_status status;
+	/** @termination: completion struct that tracks terminations */
+	struct completion termination;
+	/** @queue_list: list of exec_queues that use PXP */
+	struct list_head queue_list;
 };
 
 #endif /* __XE_PXP_TYPES_H__ */
-- 
2.43.0

[RFC 09/14] drm/xe/pxp: Add API to mark a queue as using PXP

[Daniele Ceraolo Spurio]

Userspace is required to mark a queue as using PXP to guarantee that the
PXP instructions will work. When a PXP queue is created, the driver will
do the following:
- make sure that the default PXP session is running
- set the relevant bits in the context control register
- assign an rpm ref to the queue to keep for its lifetime (this is
  required because PXP session are killed by the HW suspend flow).

When a PXP invalidation occurs, all the PXP queue will be killed.
On submission of a valid PXP queue, the driver will validate all
encrypted objects mapped to the VM to ensured they were encrypted with
the current key.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/regs/xe_engine_regs.h |  1 +
 drivers/gpu/drm/xe/xe_exec_queue.c       | 45 ++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_exec_queue.h       |  5 +++
 drivers/gpu/drm/xe/xe_hw_engine.c        |  2 +-
 drivers/gpu/drm/xe/xe_lrc.c              | 14 ++++++--
 drivers/gpu/drm/xe/xe_lrc.h              |  4 ++-
 drivers/gpu/drm/xe/xe_pxp.c              | 37 +++++++++++++++----
 drivers/gpu/drm/xe/xe_pxp.h              |  1 +
 drivers/gpu/drm/xe/xe_pxp_submit.c       |  2 +-
 include/uapi/drm/xe_drm.h                | 40 ++++++++++++++++++++-
 10 files changed, 136 insertions(+), 15 deletions(-)

diff --git a/drivers/gpu/drm/xe/regs/xe_engine_regs.h b/drivers/gpu/drm/xe/regs/xe_engine_regs.h
index c38db2a74614..0bdbda4369c5 100644
--- a/drivers/gpu/drm/xe/regs/xe_engine_regs.h
+++ b/drivers/gpu/drm/xe/regs/xe_engine_regs.h
@@ -129,6 +129,7 @@
 #define RING_EXECLIST_STATUS_HI(base)		XE_REG((base) + 0x234 + 4)
 
 #define RING_CONTEXT_CONTROL(base)		XE_REG((base) + 0x244, XE_REG_OPTION_MASKED)
+#define	  CTX_CTRL_PXP_ENABLE			REG_BIT(10)
 #define	  CTX_CTRL_OAC_CONTEXT_ENABLE		REG_BIT(8)
 #define	  CTX_CTRL_RUN_ALONE			REG_BIT(7)
 #define	  CTX_CTRL_INDIRECT_RING_STATE_ENABLE	REG_BIT(4)
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
index bc6e867aba17..ed0c49847a33 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.c
+++ b/drivers/gpu/drm/xe/xe_exec_queue.c
@@ -22,6 +22,8 @@
 #include "xe_ring_ops_types.h"
 #include "xe_trace.h"
 #include "xe_vm.h"
+#include "xe_pxp.h"
+#include "xe_pxp_types.h"
 
 enum xe_exec_queue_sched_prop {
 	XE_EXEC_QUEUE_JOB_TIMEOUT = 0,
@@ -35,6 +37,8 @@ static int exec_queue_user_extensions(struct xe_device *xe, struct xe_exec_queue
 
 static void __xe_exec_queue_free(struct xe_exec_queue *q)
 {
+	if (xe_exec_queue_uses_pxp(q))
+		xe_pxp_exec_queue_remove(gt_to_xe(q->gt)->pxp, q);
 	if (q->vm)
 		xe_vm_put(q->vm);
 	kfree(q);
@@ -69,6 +73,7 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,
 	q->ops = gt->exec_queue_ops;
 	INIT_LIST_HEAD(&q->lr.link);
 	INIT_LIST_HEAD(&q->multi_gt_link);
+	INIT_LIST_HEAD(&q->pxp.link);
 
 	q->sched_props.timeslice_us = hwe->eclass->sched_props.timeslice_us;
 	q->sched_props.preempt_timeout_us =
@@ -102,9 +107,22 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,
 static int __xe_exec_queue_init(struct xe_exec_queue *q)
 {
 	int i, err;
+	u32 flags = 0;
+
+	/*
+	 * Starting on Xe2, PXP must be enabled in the lrc. On older gens, PXP
+	 * execution on render and compute requires instead the runalone option.
+	 */
+	if (xe_exec_queue_uses_pxp(q)) {
+		if (GRAPHICS_VER(gt_to_xe(q->gt)) >= 20)
+			flags |= XE_LRC_CREATE_PXP;
+		else if ((q->class == XE_ENGINE_CLASS_RENDER ||
+			  q->class == XE_ENGINE_CLASS_COMPUTE))
+			flags |= XE_LRC_CREATE_RUNALONE;
+	}
 
 	for (i = 0; i < q->width; ++i) {
-		q->lrc[i] = xe_lrc_create(q->hwe, q->vm, SZ_16K);
+		q->lrc[i] = xe_lrc_create(q->hwe, q->vm, SZ_16K, flags);
 		if (IS_ERR(q->lrc[i])) {
 			err = PTR_ERR(q->lrc[i]);
 			goto err_lrc;
@@ -188,6 +206,9 @@ void xe_exec_queue_destroy(struct kref *ref)
 	struct xe_exec_queue *q = container_of(ref, struct xe_exec_queue, refcount);
 	struct xe_exec_queue *eq, *next;
 
+	if (xe_exec_queue_uses_pxp(q))
+		xe_pxp_exec_queue_remove(gt_to_xe(q->gt)->pxp, q);
+
 	xe_exec_queue_last_fence_put_unlocked(q);
 	if (!(q->flags & EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD)) {
 		list_for_each_entry_safe(eq, next, &q->multi_gt_list,
@@ -334,6 +355,24 @@ static int exec_queue_set_timeslice(struct xe_device *xe, struct xe_exec_queue *
 	return 0;
 }
 
+static int
+exec_queue_set_pxp_type(struct xe_device *xe, struct xe_exec_queue *q, u64 value)
+{
+	BUILD_BUG_ON(DRM_XE_PXP_TYPE_NONE != 0);
+
+	if (value == DRM_XE_PXP_TYPE_NONE)
+		return 0;
+
+	if (!xe_pxp_is_enabled(xe->pxp))
+		return -ENODEV;
+
+	/* we only support HWDRM sessions right now */
+	if (XE_IOCTL_DBG(xe, value != DRM_XE_PXP_TYPE_HWDRM))
+		return -EINVAL;
+
+	return xe_pxp_exec_queue_add(xe->pxp, q, DRM_XE_PXP_TYPE_HWDRM);
+}
+
 typedef int (*xe_exec_queue_set_property_fn)(struct xe_device *xe,
 					     struct xe_exec_queue *q,
 					     u64 value);
@@ -341,6 +380,7 @@ typedef int (*xe_exec_queue_set_property_fn)(struct xe_device *xe,
 static const xe_exec_queue_set_property_fn exec_queue_set_property_funcs[] = {
 	[DRM_XE_EXEC_QUEUE_SET_PROPERTY_PRIORITY] = exec_queue_set_priority,
 	[DRM_XE_EXEC_QUEUE_SET_PROPERTY_TIMESLICE] = exec_queue_set_timeslice,
+	[DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE] = exec_queue_set_pxp_type,
 };
 
 static int exec_queue_user_ext_set_property(struct xe_device *xe,
@@ -360,7 +400,8 @@ static int exec_queue_user_ext_set_property(struct xe_device *xe,
 			 ARRAY_SIZE(exec_queue_set_property_funcs)) ||
 	    XE_IOCTL_DBG(xe, ext.pad) ||
 	    XE_IOCTL_DBG(xe, ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_PRIORITY &&
-			 ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_TIMESLICE))
+			 ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_TIMESLICE &&
+			 ext.property != DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE))
 		return -EINVAL;
 
 	idx = array_index_nospec(ext.property, ARRAY_SIZE(exec_queue_set_property_funcs));
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.h b/drivers/gpu/drm/xe/xe_exec_queue.h
index 289a3a51d2a2..572e5f10a342 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.h
+++ b/drivers/gpu/drm/xe/xe_exec_queue.h
@@ -53,6 +53,11 @@ static inline bool xe_exec_queue_is_parallel(struct xe_exec_queue *q)
 	return q->width > 1;
 }
 
+static inline bool xe_exec_queue_uses_pxp(struct xe_exec_queue *q)
+{
+	return q->pxp.type;
+}
+
 bool xe_exec_queue_is_lr(struct xe_exec_queue *q);
 
 bool xe_exec_queue_ring_full(struct xe_exec_queue *q);
diff --git a/drivers/gpu/drm/xe/xe_hw_engine.c b/drivers/gpu/drm/xe/xe_hw_engine.c
index 07ed9fd28f19..572b7572cd7a 100644
--- a/drivers/gpu/drm/xe/xe_hw_engine.c
+++ b/drivers/gpu/drm/xe/xe_hw_engine.c
@@ -528,7 +528,7 @@ static int hw_engine_init(struct xe_gt *gt, struct xe_hw_engine *hwe,
 		goto err_name;
 	}
 
-	hwe->kernel_lrc = xe_lrc_create(hwe, NULL, SZ_16K);
+	hwe->kernel_lrc = xe_lrc_create(hwe, NULL, SZ_16K, 0);
 	if (IS_ERR(hwe->kernel_lrc)) {
 		err = PTR_ERR(hwe->kernel_lrc);
 		goto err_hwsp;
diff --git a/drivers/gpu/drm/xe/xe_lrc.c b/drivers/gpu/drm/xe/xe_lrc.c
index 94ff62e1d95e..81c179012e3a 100644
--- a/drivers/gpu/drm/xe/xe_lrc.c
+++ b/drivers/gpu/drm/xe/xe_lrc.c
@@ -890,7 +890,7 @@ static void xe_lrc_finish(struct xe_lrc *lrc)
 #define PVC_CTX_ACC_CTR_THOLD	(0x2a + 1)
 
 static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
-		       struct xe_vm *vm, u32 ring_size)
+		       struct xe_vm *vm, u32 ring_size, u32 init_flags)
 {
 	struct xe_gt *gt = hwe->gt;
 	struct xe_tile *tile = gt_to_tile(gt);
@@ -978,6 +978,14 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
 				     RING_CTL_SIZE(lrc->ring.size) | RING_VALID);
 	}
 
+	if (init_flags & XE_LRC_CREATE_RUNALONE)
+		xe_lrc_write_ctx_reg(lrc, CTX_CONTEXT_CONTROL,
+				     _MASKED_BIT_ENABLE(CTX_CTRL_RUN_ALONE));
+
+	if (init_flags & XE_LRC_CREATE_PXP)
+		xe_lrc_write_ctx_reg(lrc, CTX_CONTEXT_CONTROL,
+				     _MASKED_BIT_ENABLE(CTX_CTRL_PXP_ENABLE));
+
 	xe_lrc_write_ctx_reg(lrc, CTX_TIMESTAMP, 0);
 
 	if (xe->info.has_asid && vm)
@@ -1026,7 +1034,7 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
  * upon failure.
  */
 struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
-			     u32 ring_size)
+			     u32 ring_size, u32 flags)
 {
 	struct xe_lrc *lrc;
 	int err;
@@ -1035,7 +1043,7 @@ struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
 	if (!lrc)
 		return ERR_PTR(-ENOMEM);
 
-	err = xe_lrc_init(lrc, hwe, vm, ring_size);
+	err = xe_lrc_init(lrc, hwe, vm, ring_size, flags);
 	if (err) {
 		kfree(lrc);
 		return ERR_PTR(err);
diff --git a/drivers/gpu/drm/xe/xe_lrc.h b/drivers/gpu/drm/xe/xe_lrc.h
index d411c3fbcbc6..cc8091bba2a0 100644
--- a/drivers/gpu/drm/xe/xe_lrc.h
+++ b/drivers/gpu/drm/xe/xe_lrc.h
@@ -23,8 +23,10 @@ struct xe_vm;
 #define LRC_PPHWSP_FLUSH_INVAL_SCRATCH_ADDR (0x34 * 4)
 #define LRC_PPHWSP_PXP_INVAL_SCRATCH_ADDR (0x40 * 4)
 
+#define XE_LRC_CREATE_RUNALONE 0x1
+#define XE_LRC_CREATE_PXP 0x2
 struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
-			     u32 ring_size);
+			     u32 ring_size, u32 flags);
 void xe_lrc_destroy(struct kref *ref);
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index 159761e19e9b..e979b47f155a 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -6,10 +6,13 @@
 #include "xe_pxp.h"
 
 #include <drm/drm_managed.h>
+#include <drm/xe_drm.h>
 
 #include "xe_device_types.h"
+#include "xe_exec_queue.h"
 #include "xe_exec_queue_types.h"
 #include "xe_force_wake.h"
+#include "xe_guc_submit.h"
 #include "xe_gsc_proxy.h"
 #include "xe_gt.h"
 #include "xe_gt_types.h"
@@ -30,14 +33,14 @@
  * integrated parts.
  */
 
-#define ARB_SESSION 0xF /* TODO: move to UAPI */
+#define ARB_SESSION DRM_XE_PXP_HWDRM_DEFAULT_SESSION /* shorter define */
 
 bool xe_pxp_is_supported(const struct xe_device *xe)
 {
 	return xe->info.has_pxp;
 }
 
-static bool pxp_is_enabled(const struct xe_pxp *pxp)
+bool xe_pxp_is_enabled(const struct xe_pxp *pxp)
 {
 	return pxp;
 }
@@ -86,12 +89,16 @@ static int pxp_wait_for_session_state(struct xe_pxp *pxp, u32 id, bool in_play)
 	return ret;
 }
 
+static void pxp_invalidate_queues(struct xe_pxp *pxp);
+
 static void pxp_terminate(struct xe_pxp *pxp)
 {
 	int ret = 0;
 	struct xe_device *xe = pxp->xe;
 	struct xe_gt *gt = pxp->gt;
 
+	pxp_invalidate_queues(pxp);
+
 	/*
 	 * If we have a termination already in progress, we need to wait for
 	 * it to complete before queueing another one. We update the state
@@ -202,7 +209,7 @@ void xe_pxp_irq_handler(struct xe_device *xe, u16 iir)
 {
 	struct xe_pxp *pxp = xe->pxp;
 
-	if (!pxp_is_enabled(pxp)) {
+	if (!xe_pxp_is_enabled(pxp)) {
 		drm_err(&xe->drm, "PXP irq 0x%x received with PXP disabled!\n", iir);
 		return;
 	}
@@ -385,9 +392,12 @@ int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type)
 {
 	int ret = 0;
 
-	if (!pxp_is_enabled(pxp))
+	if (!xe_pxp_is_enabled(pxp))
 		return -ENODEV;
 
+	/* we only support HWDRM sessions right now */
+	xe_assert(pxp->xe, type == DRM_XE_PXP_TYPE_HWDRM);
+
 	/*
 	 * Runtime suspend kills PXP, so we need to turn it off while we have
 	 * active queues that use PXP
@@ -472,14 +482,14 @@ void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q)
 {
 	bool need_pm_put = false;
 
-	if (!pxp_is_enabled(pxp))
+	if (!xe_pxp_is_enabled(pxp))
 		return;
 
 	mutex_lock(&pxp->mutex);
 
 	if (!list_empty(&q->pxp.link)) {
 		list_del_init(&q->pxp.link);
-		q->pxp.type = 0;
+		q->pxp.type = DRM_XE_PXP_TYPE_NONE;
 		need_pm_put = true;
 	}
 
@@ -488,3 +498,18 @@ void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q)
 	if (need_pm_put)
 		xe_pm_runtime_put(pxp->xe);
 }
+
+static void pxp_invalidate_queues(struct xe_pxp *pxp)
+{
+	struct xe_exec_queue *tmp, *q;
+
+	list_for_each_entry(tmp, &pxp->queue_list,  pxp.link) {
+		q = xe_exec_queue_get_unless_zero(tmp);
+
+		if (!q)
+			continue;
+
+		xe_exec_queue_kill(q);
+		xe_exec_queue_put(q);
+	}
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index 6c99af072ab0..8f0a3a514fb8 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -13,6 +13,7 @@ struct xe_exec_queue;
 struct xe_pxp;
 
 bool xe_pxp_is_supported(const struct xe_device *xe);
+bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
 
 int xe_pxp_init(struct xe_device *xe);
 void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
index 9c76eeb5535c..1ad0273fde77 100644
--- a/drivers/gpu/drm/xe/xe_pxp_submit.c
+++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
@@ -473,7 +473,7 @@ int xe_pxp_submit_session_init(struct xe_pxp *pxp, int id)
 				   FIELD_PREP(PXP43_INIT_SESSION_APPTYPE, 0));
 	msg_in.header.buffer_len = sizeof(msg_in) - sizeof(msg_in.header);
 
-	if (id == ARB_SESSION)
+	if (id == DRM_XE_PXP_HWDRM_DEFAULT_SESSION)
 		msg_in.protection_mode = PXP43_INIT_SESSION_PROTECTION_ARB;
 
 	ret = gsccs_send_message(pxp, &msg_in, sizeof(msg_in), &msg_out, sizeof(msg_out));
diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
index 12eaa8532b5c..631fdc2ed493 100644
--- a/include/uapi/drm/xe_drm.h
+++ b/include/uapi/drm/xe_drm.h
@@ -1071,6 +1071,24 @@ struct drm_xe_vm_bind {
 /**
  * struct drm_xe_exec_queue_create - Input of &DRM_IOCTL_XE_EXEC_QUEUE_CREATE
  *
+ * This ioctl supports setting the following properties via the
+ * %DRM_XE_EXEC_QUEUE_EXTENSION_SET_PROPERTY extension, which uses the the
+ * generic @drm_xe_ext_set_property struct:
+ *
+ *  - %DRM_XE_EXEC_QUEUE_SET_PROPERTY_PRIORITY - set the queue priority.
+ *    CAP_SYS_NICE is required to set a value above normal.
+ *  - %DRM_XE_EXEC_QUEUE_SET_PROPERTY_TIMESLICE - set the queue timeslice
+ *    duration.
+ *  - %DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE - set the type of PXP session
+ *    this queue will be used with. Valid values are listed in enum
+ *    drm_xe_pxp_session_type. %DRM_XE_PXP_TYPE_NONE is the default behavior, so
+ *    there is no need to explicitly set that. When a queue of type
+ *    %DRM_XE_PXP_TYPE_HWDRM is created, the PXP default HWDRM session
+ *    (%XE_PXP_HWDRM_DEFAULT_SESSION) will be started, if isn't already running.
+ *    Given that going into a power-saving state kills PXP HWDRM sessions,
+ *    runtime PM will be blocked while queues of this type are alive.
+ *    All PXP queues will be killed if a PXP invalidation event occurs.
+ *
  * The example below shows how to use @drm_xe_exec_queue_create to create
  * a simple exec_queue (no parallel submission) of class
  * &DRM_XE_ENGINE_CLASS_RENDER.
@@ -1094,7 +1112,7 @@ struct drm_xe_exec_queue_create {
 #define DRM_XE_EXEC_QUEUE_EXTENSION_SET_PROPERTY		0
 #define   DRM_XE_EXEC_QUEUE_SET_PROPERTY_PRIORITY		0
 #define   DRM_XE_EXEC_QUEUE_SET_PROPERTY_TIMESLICE		1
-
+#define   DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE		2
 	/** @extensions: Pointer to the first extension struct, if any */
 	__u64 extensions;
 
@@ -1678,6 +1696,26 @@ struct drm_xe_oa_stream_info {
 	__u64 reserved[3];
 };
 
+/**
+ * enum drm_xe_pxp_session_type - Supported PXP session types.
+ *
+ * We currently only support HWDRM sessions, which are used for protected
+ * content that ends up being displayed, but the HW supports multiple types, so
+ * we might extend support in the future.
+ */
+enum drm_xe_pxp_session_type {
+	/** @DRM_XE_PXP_TYPE_NONE: PXP not used */
+	DRM_XE_PXP_TYPE_NONE = 0,
+	/**
+	 * @DRM_XE_PXP_TYPE_HWDRM: HWDRM sessions are used for content that ends
+	 * up on the display.
+	 */
+	DRM_XE_PXP_TYPE_HWDRM
+};
+
+/* ID of the protected content session managed by Xe when PXP is active */
+#define DRM_XE_PXP_HWDRM_DEFAULT_SESSION 0xf
+
 #if defined(__cplusplus)
 }
 #endif
-- 
2.43.0

[RFC 10/14] drm/xe/pxp: add a query for PXP status

[Daniele Ceraolo Spurio]

PXP prerequisites (SW proxy and HuC auth via GSC) are completed
asynchronously from driver load, which means that userspace can start
submitting before we're ready to start a PXP session. Therefore, we need
a query that userspace can use to check not only if PXP is supported by
also to wait until the prerequisites are done.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp.h   |  1 +
 drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
 include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
 4 files changed, 95 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index e979b47f155a..e39a47aeb050 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
 	return ready;
 }
 
+/**
+ * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
+ * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
+ *
+ * This function is used for status query from userspace, so the returned value
+ * follow the uapi (see drm_xe_query_pxp_status)
+ *
+ * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
+ * is not supported/enabled or if something went wrong in the initialization of
+ * the prerequisites.
+ */
+int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
+{
+	int ret = 0;
+
+	if (!xe_pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	/* if the GSC or HuC FW are in an error state, PXP will never work */
+	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
+	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
+		return -EIO;
+
+	xe_pm_runtime_get(pxp->xe);
+
+	/* PXP requires both HuC loaded and GSC proxy initialized */
+	if (pxp_prerequisites_done(pxp))
+		ret = 1;
+
+	xe_pm_runtime_put(pxp->xe);
+	return ret;
+}
+
 static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
 {
 	struct xe_gt *gt = pxp->gt;
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index 8f0a3a514fb8..b2aae4abdd0e 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -14,6 +14,7 @@ struct xe_pxp;
 
 bool xe_pxp_is_supported(const struct xe_device *xe);
 bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
+int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
 
 int xe_pxp_init(struct xe_device *xe);
 void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
index 4e01df6b1b7a..5da9f403c2b9 100644
--- a/drivers/gpu/drm/xe/xe_query.c
+++ b/drivers/gpu/drm/xe/xe_query.c
@@ -22,6 +22,7 @@
 #include "xe_guc_hwconfig.h"
 #include "xe_macros.h"
 #include "xe_mmio.h"
+#include "xe_pxp.h"
 #include "xe_ttm_vram_mgr.h"
 
 static const u16 xe_to_user_engine_class[] = {
@@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
 	return ret ? -EFAULT : 0;
 }
 
+static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
+{
+	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
+	size_t size = sizeof(struct drm_xe_query_pxp_status);
+	struct drm_xe_query_pxp_status resp;
+	int ret;
+
+	if (query->size == 0) {
+		query->size = size;
+		return 0;
+	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
+		return -EINVAL;
+	}
+
+	if (copy_from_user(&resp, query_ptr, size))
+		return -EFAULT;
+
+	ret = xe_pxp_get_readiness_status(xe->pxp);
+	if (ret < 0)
+		return ret;
+
+	resp.status = ret;
+	resp.supported_session_types =
+		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
+
+	if (copy_to_user(query_ptr, &resp, size))
+		 return -EFAULT;
+
+	return 0;
+}
+
 static int (* const xe_query_funcs[])(struct xe_device *xe,
 				      struct drm_xe_device_query *query) = {
 	query_engines,
@@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
 	query_engine_cycles,
 	query_uc_fw_version,
 	query_oa_units,
+	query_pxp_status,
 };
 
 int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
index 631fdc2ed493..746bc16bd220 100644
--- a/include/uapi/drm/xe_drm.h
+++ b/include/uapi/drm/xe_drm.h
@@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
 	__u64 reserved;
 };
 
+/**
+ * struct drm_xe_query_pxp_status - query if PXP is ready
+ *
+ * If PXP is enabled and no fatal error as occurred, the status will be set to
+ * one of the following values:
+ * 0: PXP init still in progress
+ * 1: PXP init complete
+ *
+ * The supported session type bitmask is based on the values in
+ * enum drm_xe_pxp_session_type, including TYPE_NONE.
+ *
+ * If PXP is not enabled or something has gone wrong, the query will be failed
+ * with one of the following error codes:
+ * -ENODEV: PXP not supported or disabled;
+ * -EIO: fatal error occurred during init, so PXP will never be enabled;
+ * -EINVAL: incorrect value provided as part of the query;
+ * -EFAULT: error copying the memory between kernel and userspace.
+ */
+struct drm_xe_query_pxp_status {
+	/** @status: current PXP status */
+	__u32 status;
+
+	/** @supported_session_types: bitmask of supported PXP session types */
+	__u32 supported_session_types;
+};
+
 /**
  * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
  * structure to query device information
@@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
  *    attributes.
  *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
  *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
+ *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
  *
  * If size is set to 0, the driver fills it with the required size for
  * the requested type of data to query. If size is equal to the required
@@ -690,6 +717,7 @@ struct drm_xe_device_query {
 #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
 #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
 #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
+#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
 	/** @query: The type of data to query */
 	__u32 query;
 
-- 
2.43.0

[RFC 11/14] drm/xe/pxp: Add API to mark a BO as using PXP

[Daniele Ceraolo Spurio]

The driver needs to know if a BO is encrypted with PXP to enable the
display decription at flip time.
Furthermore, we want to keep track of the status of the encryption and
reject any operation that involves a BO that is encrypted using an old
key.

RFC note: the rejection at exec time is done by looping through all the
BOs mapped in the VM and checking all of them. The alternative would be
to keep a dedicated list of PXP BO in the VM struct instead and only
check the BOs in that list.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 .../xe/compat-i915-headers/pxp/intel_pxp.h    |  10 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 100 +++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   5 +
 drivers/gpu/drm/xe/xe_bo_types.h              |   3 +
 drivers/gpu/drm/xe/xe_exec.c                  |   6 ++
 drivers/gpu/drm/xe/xe_pxp.c                   |  74 +++++++++++++
 drivers/gpu/drm/xe/xe_pxp.h                   |   4 +
 drivers/gpu/drm/xe/xe_pxp_types.h             |   3 +
 drivers/gpu/drm/xe/xe_vm.c                    |  42 +++++++-
 drivers/gpu/drm/xe/xe_vm.h                    |   2 +
 include/uapi/drm/xe_drm.h                     |  15 +++
 11 files changed, 258 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h b/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
index 881680727452..d8682f781619 100644
--- a/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
+++ b/drivers/gpu/drm/xe/compat-i915-headers/pxp/intel_pxp.h
@@ -9,6 +9,9 @@
 #include <linux/errno.h>
 #include <linux/types.h>
 
+#include "xe_bo.h"
+#include "xe_pxp.h"
+
 struct drm_i915_gem_object;
 struct xe_pxp;
 
@@ -16,13 +19,16 @@ static inline int intel_pxp_key_check(struct xe_pxp *pxp,
 				      struct drm_i915_gem_object *obj,
 				      bool assign)
 {
-	return -ENODEV;
+	if (assign)
+		return -EINVAL;
+
+	return xe_pxp_key_check(pxp, obj);
 }
 
 static inline bool
 i915_gem_object_is_protected(const struct drm_i915_gem_object *obj)
 {
-	return false;
+	return xe_bo_is_protected(obj);
 }
 
 #endif
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 65c696966e96..90ee91444ce5 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -6,6 +6,7 @@
 #include "xe_bo.h"
 
 #include <linux/dma-buf.h>
+#include <linux/nospec.h>
 
 #include <drm/drm_drv.h>
 #include <drm/drm_gem_ttm_helper.h>
@@ -24,6 +25,7 @@
 #include "xe_migrate.h"
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
+#include "xe_pxp.h"
 #include "xe_res_cursor.h"
 #include "xe_trace_bo.h"
 #include "xe_ttm_stolen_mgr.h"
@@ -1931,6 +1933,95 @@ void xe_bo_vunmap(struct xe_bo *bo)
 	__xe_bo_vunmap(bo);
 }
 
+static int gem_create_set_pxp_type(struct xe_device *xe, struct xe_bo *bo, u64 value)
+{
+	if (value == DRM_XE_PXP_TYPE_NONE)
+		return 0;
+
+	/* we only support DRM_XE_PXP_TYPE_HWDRM for now */
+	if (XE_IOCTL_DBG(xe, value != DRM_XE_PXP_TYPE_HWDRM))
+		return -EINVAL;
+
+	xe_pxp_key_assign(xe->pxp, bo);
+
+	return 0;
+}
+
+typedef int (*xe_gem_create_set_property_fn)(struct xe_device *xe,
+					     struct xe_bo *bo,
+					     u64 value);
+
+static const xe_gem_create_set_property_fn gem_create_set_property_funcs[] = {
+	[DRM_XE_GEM_CREATE_EXTENSION_SET_PROPERTY] = gem_create_set_pxp_type,
+};
+
+static int gem_create_user_ext_set_property(struct xe_device *xe,
+					    struct xe_bo *bo,
+					    u64 extension)
+{
+	u64 __user *address = u64_to_user_ptr(extension);
+	struct drm_xe_ext_set_property ext;
+	int err;
+	u32 idx;
+
+	err = __copy_from_user(&ext, address, sizeof(ext));
+	if (XE_IOCTL_DBG(xe, err))
+		return -EFAULT;
+
+	if (XE_IOCTL_DBG(xe, ext.property >=
+			 ARRAY_SIZE(gem_create_set_property_funcs)) ||
+	    XE_IOCTL_DBG(xe, ext.pad) ||
+	    XE_IOCTL_DBG(xe, ext.property != DRM_XE_GEM_CREATE_EXTENSION_SET_PROPERTY))
+		return -EINVAL;
+
+	idx = array_index_nospec(ext.property, ARRAY_SIZE(gem_create_set_property_funcs));
+	if (!gem_create_set_property_funcs[idx])
+		return -EINVAL;
+
+	return gem_create_set_property_funcs[idx](xe, bo, ext.value);
+}
+
+typedef int (*xe_gem_create_user_extension_fn)(struct xe_device *xe,
+					       struct xe_bo *bo,
+					       u64 extension);
+
+static const xe_gem_create_user_extension_fn gem_create_user_extension_funcs[] = {
+	[DRM_XE_GEM_CREATE_EXTENSION_SET_PROPERTY] = gem_create_user_ext_set_property,
+};
+
+#define MAX_USER_EXTENSIONS	16
+static int gem_create_user_extensions(struct xe_device *xe, struct xe_bo *bo,
+				      u64 extensions, int ext_number)
+{
+	u64 __user *address = u64_to_user_ptr(extensions);
+	struct drm_xe_user_extension ext;
+	int err;
+	u32 idx;
+
+	if (XE_IOCTL_DBG(xe, ext_number >= MAX_USER_EXTENSIONS))
+		return -E2BIG;
+
+	err = __copy_from_user(&ext, address, sizeof(ext));
+	if (XE_IOCTL_DBG(xe, err))
+		return -EFAULT;
+
+	if (XE_IOCTL_DBG(xe, ext.pad) ||
+	    XE_IOCTL_DBG(xe, ext.name >= ARRAY_SIZE(gem_create_user_extension_funcs)))
+		return -EINVAL;
+
+	idx = array_index_nospec(ext.name,
+				 ARRAY_SIZE(gem_create_user_extension_funcs));
+	err = gem_create_user_extension_funcs[idx](xe, bo, extensions);
+	if (XE_IOCTL_DBG(xe, err))
+		return err;
+
+	if (ext.next_extension)
+		return gem_create_user_extensions(xe, bo, ext.next_extension,
+						  ++ext_number);
+
+	return 0;
+}
+
 int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			struct drm_file *file)
 {
@@ -1943,8 +2034,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	u32 handle;
 	int err;
 
-	if (XE_IOCTL_DBG(xe, args->extensions) ||
-	    XE_IOCTL_DBG(xe, args->pad[0] || args->pad[1] || args->pad[2]) ||
+	if (XE_IOCTL_DBG(xe, args->pad[0] || args->pad[1] || args->pad[2]) ||
 	    XE_IOCTL_DBG(xe, args->reserved[0] || args->reserved[1]))
 		return -EINVAL;
 
@@ -2019,6 +2109,12 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		goto out_vm;
 	}
 
+	if (args->extensions) {
+		err = gem_create_user_extensions(xe, bo, args->extensions, 0);
+		if(err)
+			goto out_bulk;
+	}
+
 	err = drm_gem_handle_create(file, &bo->ttm.base, &handle);
 	if (err)
 		goto out_bulk;
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 6de894c728f5..0bf0b077f063 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -170,6 +170,11 @@ static inline bool xe_bo_is_pinned(struct xe_bo *bo)
 	return bo->ttm.pin_count;
 }
 
+static inline bool xe_bo_is_protected(const struct xe_bo *bo)
+{
+    return bo->pxp_key_instance;
+}
+
 static inline void xe_bo_unpin_map_no_vm(struct xe_bo *bo)
 {
 	if (likely(bo)) {
diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index 86422e113d39..d4f67180b413 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -56,6 +56,9 @@ struct xe_bo {
 	 */
 	struct list_head client_link;
 #endif
+	/** @pxp_key_instance: key instance this bo was created against (if any) */
+	u32 pxp_key_instance;
+
 	/** @freed: List node for delayed put. */
 	struct llist_node freed;
 	/** @created: Whether the bo has passed initial creation */
diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c
index 2d72cdec3a0b..933da5b9f59c 100644
--- a/drivers/gpu/drm/xe/xe_exec.c
+++ b/drivers/gpu/drm/xe/xe_exec.c
@@ -250,6 +250,12 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		goto err_exec;
 	}
 
+	if (xe_exec_queue_uses_pxp(q)) {
+		err = xe_vm_validate_protected(q->vm);
+		if (err)
+			goto err_exec;
+	}
+
 	job = xe_sched_job_create(q, xe_exec_queue_is_parallel(q) ?
 				  addresses : &args->address);
 	if (IS_ERR(job)) {
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index e39a47aeb050..493b8ff4b743 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -8,6 +8,8 @@
 #include <drm/drm_managed.h>
 #include <drm/xe_drm.h>
 
+#include "xe_bo.h"
+#include "xe_bo_types.h"
 #include "xe_device_types.h"
 #include "xe_exec_queue.h"
 #include "xe_exec_queue_types.h"
@@ -132,6 +134,9 @@ static void pxp_terminate(struct xe_pxp *pxp)
 
 	pxp_invalidate_queues(pxp);
 
+	if (pxp->status == XE_PXP_ACTIVE)
+		pxp->key_instance++;
+
 	/*
 	 * If we have a termination already in progress, we need to wait for
 	 * it to complete before queueing another one. We update the state
@@ -344,6 +349,8 @@ int xe_pxp_init(struct xe_device *xe)
 	pxp->xe = xe;
 	pxp->gt = gt;
 
+	pxp->key_instance = 1;
+
 	/*
 	 * we'll use the completion to check if there is a termination pending,
 	 * so we start it as completed and we reinit it when a termination
@@ -546,3 +553,70 @@ static void pxp_invalidate_queues(struct xe_pxp *pxp)
 		xe_exec_queue_put(q);
 	}
 }
+
+/**
+ * xe_pxp_key_assign - mark a BO as using the current PXP key iteration
+ * @pxp: the xe->pxp pointer (it will be NULL if PXP is disabled)
+ * @bo: the BO to mark
+ *
+ * Returns: -ENODEV if PXP is disabled, 0 otherwise.
+ */
+int xe_pxp_key_assign(struct xe_pxp *pxp, struct xe_bo *bo)
+{
+	if (!xe_pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	xe_assert(pxp->xe, !bo->pxp_key_instance);
+
+	/*
+	 * Note that the PXP key handling is inherently racey, because the key
+	 * can theoretically change at any time (although it's unlikely to do
+	 * so without triggers), even right after we copy it. Taking a lock
+	 * wouldn't help because the value might still change as soon as we
+	 * release the lock.
+	 * Userspace needs to handle the fact that their BOs can go invalid at
+	 * any point.
+	 */
+	bo->pxp_key_instance = pxp->key_instance;
+
+	return 0;
+}
+
+/**
+ * xe_pxp_key_check - check if the key used by a BO is valid
+ * @pxp: the xe->pxp pointer (it will be NULL if PXP is disabled)
+ * @bo: the BO we want to check
+ *
+ * Checks whether a BO was encrypted with the current key or an obsolete one.
+ *
+ * Returns: 0 if the key is valid, -ENODEV if PXP is disabled, -EINVAL if the
+ * BO is not using PXP,  -ENOEXEC if the key is not valid.
+ */
+int xe_pxp_key_check(struct xe_pxp *pxp, struct xe_bo *bo)
+{
+	if (!xe_pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	if (!xe_bo_is_protected(bo))
+		return -EINVAL;
+
+	xe_assert(pxp->xe, bo->pxp_key_instance);
+
+	/*
+	 * Note that the PXP key handling is inherently racey, because the key
+	 * can theoretically change at any time (although it's unlikely to do
+	 * so without triggers), even right after we check it. Taking a lock
+	 * wouldn't help because the value might still change as soon as we
+	 * release the lock.
+	 * We mitigate the risk by checking the key at multiple points (on each
+	 * submission involving the BO and right before flipping it on the
+	 * display), but there is still a very small chance that we could
+	 * operate on an invalid BO for a single submission or a single frame
+	 * flip. This is a compromise made to protect the encrypted data (which
+	 * is what the key termination is for).
+	 */
+	if (bo->pxp_key_instance != pxp->key_instance)
+		return -ENOEXEC;
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index b2aae4abdd0e..e3c4c92fc7a3 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -8,6 +8,7 @@
 
 #include <linux/types.h>
 
+struct xe_bo;
 struct xe_device;
 struct xe_exec_queue;
 struct xe_pxp;
@@ -22,4 +23,7 @@ void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
 int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type);
 void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q);
 
+int xe_pxp_key_assign(struct xe_pxp *pxp, struct xe_bo *bo);
+int xe_pxp_key_check(struct xe_pxp *pxp, struct xe_bo *bo);
+
 #endif /* __XE_PXP_H__ */
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
index 5f7b031cafb1..e28bcbda1ec6 100644
--- a/drivers/gpu/drm/xe/xe_pxp_types.h
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -88,6 +88,9 @@ struct xe_pxp {
 	struct completion termination;
 	/** @queue_list: list of exec_queues that use PXP */
 	struct list_head queue_list;
+
+	/** @key_instance: keep track of the current iteration of the PXP key */
+	u32 key_instance;
 };
 
 #endif /* __XE_PXP_TYPES_H__ */
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 412ec9cb9650..f7be115b04cf 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -34,6 +34,7 @@
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
 #include "xe_pt.h"
+#include "xe_pxp.h"
 #include "xe_res_cursor.h"
 #include "xe_sync.h"
 #include "xe_trace_bo.h"
@@ -3069,7 +3070,7 @@ static void xe_vma_ops_init(struct xe_vma_ops *vops, struct xe_vm *vm,
 
 static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,
 					u64 addr, u64 range, u64 obj_offset,
-					u16 pat_index)
+					u16 pat_index, u32 op)
 {
 	u16 coh_mode;
 
@@ -3104,6 +3105,12 @@ static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,
 		return  -EINVAL;
 	}
 
+	/* If a BO is protected it must be valid to be mapped */
+	if (xe_bo_is_protected(bo) &&
+	    op != DRM_XE_VM_BIND_OP_UNMAP && op != DRM_XE_VM_BIND_OP_UNMAP_ALL)
+		if (XE_IOCTL_DBG(xe, xe_pxp_key_check(xe->pxp, bo) != 0))
+			return -ENOEXEC;
+
 	return 0;
 }
 
@@ -3191,6 +3198,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		u32 obj = bind_ops[i].obj;
 		u64 obj_offset = bind_ops[i].obj_offset;
 		u16 pat_index = bind_ops[i].pat_index;
+		u32 op = bind_ops[i].op;
 
 		if (!obj)
 			continue;
@@ -3203,7 +3211,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 		bos[i] = gem_to_xe_bo(gem_obj);
 
 		err = xe_vm_bind_ioctl_validate_bo(xe, bos[i], addr, range,
-						   obj_offset, pat_index);
+						   obj_offset, pat_index, op);
 		if (err)
 			goto put_obj;
 	}
@@ -3478,6 +3486,36 @@ int xe_vm_invalidate_vma(struct xe_vma *vma)
 	return 0;
 }
 
+int xe_vm_validate_protected(struct xe_vm *vm)
+{
+	struct drm_gpuva *gpuva;
+	int err = 0;
+
+	if (!vm)
+		return -ENODEV;
+
+	mutex_lock(&vm->snap_mutex);
+
+	drm_gpuvm_for_each_va(gpuva, &vm->gpuvm) {
+		struct xe_vma *vma = gpuva_to_vma(gpuva);
+		struct xe_bo *bo = vma->gpuva.gem.obj ?
+			gem_to_xe_bo(vma->gpuva.gem.obj) : NULL;
+
+		if (!bo)
+			continue;
+
+		if (xe_bo_is_protected(bo)) {
+			err = xe_pxp_key_check(vm->xe->pxp, bo);
+			if (err)
+				break;
+		}
+	}
+
+	mutex_unlock(&vm->snap_mutex);
+	return err;
+}
+
+
 struct xe_vm_snapshot {
 	unsigned long num_snaps;
 	struct {
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index 5e298ac90dfc..fdb043507293 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -216,6 +216,8 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma,
 
 int xe_vm_invalidate_vma(struct xe_vma *vma);
 
+int xe_vm_validate_protected(struct xe_vm *vm);
+
 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm)
 {
 	xe_assert(vm->xe, xe_vm_in_preempt_fence_mode(vm));
diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
index 746bc16bd220..abec3b27ca08 100644
--- a/include/uapi/drm/xe_drm.h
+++ b/include/uapi/drm/xe_drm.h
@@ -761,8 +761,23 @@ struct drm_xe_device_query {
  *  - %DRM_XE_GEM_CPU_CACHING_WC - Allocate the pages as write-combined. This
  *    is uncached. Scanout surfaces should likely use this. All objects
  *    that can be placed in VRAM must use this.
+ *
+ * This ioctl supports setting the following properties via the
+ * %DRM_XE_GEM_CREATE_EXTENSION_SET_PROPERTY extension, which uses the the
+ * generic @drm_xe_ext_set_property struct:
+ *
+ *  - %DRM_XE_GEM_CREATE_SET_PROPERTY_PXP_TYPE - set the type of PXP session
+ *    this object will be used with. Valid values are listed in enum
+ *    drm_xe_pxp_session_type. %DRM_XE_PXP_TYPE_NONE is the default behavior, so
+ *    there is no need to explicitly set that. Objects used with session of type
+ *    %DRM_XE_PXP_TYPE_HWDRM will be marked as invalid if a PXP invalidation
+ *    event occurs after their creation. Attempting to flip an invalid object
+ *    will cause a black frame to be displayed instead. Submissions with invalid
+ *    objects mapped in the VM will be rejected.
  */
 struct drm_xe_gem_create {
+#define DRM_XE_GEM_CREATE_EXTENSION_SET_PROPERTY	0
+#define   DRM_XE_GEM_CREATE_SET_PROPERTY_PXP_TYPE	0
 	/** @extensions: Pointer to the first extension struct, if any */
 	__u64 extensions;
 
-- 
2.43.0

[RFC 12/14] drm/xe/pxp: add PXP PM support

[Daniele Ceraolo Spurio]

The HW suspend flow kills all PXP HWDRM sessions, so if there was any
PXP activity before the suspend we need to trigger a full termination on
suspend.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/xe_pm.c        | 42 +++++++++++---
 drivers/gpu/drm/xe/xe_pxp.c       | 95 ++++++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_pxp.h       |  3 +
 drivers/gpu/drm/xe/xe_pxp_types.h |  9 ++-
 4 files changed, 137 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_pm.c b/drivers/gpu/drm/xe/xe_pm.c
index de3b5df65e48..6586f73638db 100644
--- a/drivers/gpu/drm/xe/xe_pm.c
+++ b/drivers/gpu/drm/xe/xe_pm.c
@@ -20,6 +20,7 @@
 #include "xe_guc.h"
 #include "xe_irq.h"
 #include "xe_pcode.h"
+#include "xe_pxp.h"
 #include "xe_wa.h"
 
 /**
@@ -88,22 +89,24 @@ int xe_pm_suspend(struct xe_device *xe)
 
 	drm_dbg(&xe->drm, "Suspending device\n");
 
+	err = xe_pxp_pm_suspend(xe->pxp);
+	if (err)
+		goto err;
+
 	for_each_gt(gt, xe, id)
 		xe_gt_suspend_prepare(gt);
 
 	/* FIXME: Super racey... */
 	err = xe_bo_evict_all(xe);
 	if (err)
-		goto err;
+		goto err_pxp;
 
 	xe_display_pm_suspend(xe, false);
 
 	for_each_gt(gt, xe, id) {
 		err = xe_gt_suspend(gt);
-		if (err) {
-			xe_display_pm_resume(xe, false);
-			goto err;
-		}
+		if (err)
+			goto err_display;
 	}
 
 	xe_irq_suspend(xe);
@@ -112,6 +115,11 @@ int xe_pm_suspend(struct xe_device *xe)
 
 	drm_dbg(&xe->drm, "Device suspended\n");
 	return 0;
+
+err_display:
+	xe_display_pm_resume(xe, false);
+err_pxp:
+	xe_pxp_pm_resume(xe->pxp);
 err:
 	drm_dbg(&xe->drm, "Device suspend failed %d\n", err);
 	return err;
@@ -160,6 +168,8 @@ int xe_pm_resume(struct xe_device *xe)
 	if (err)
 		goto err;
 
+	xe_pxp_pm_resume(xe->pxp);
+
 	drm_dbg(&xe->drm, "Device resumed\n");
 	return 0;
 err:
@@ -352,6 +362,10 @@ int xe_pm_runtime_suspend(struct xe_device *xe)
 	 */
 	lock_map_acquire(&xe_pm_runtime_lockdep_map);
 
+	err = xe_pxp_pm_suspend(xe->pxp);
+	if (err)
+		goto out;
+
 	/*
 	 * Applying lock for entire list op as xe_ttm_bo_destroy and xe_bo_move_notify
 	 * also checks and delets bo entry from user fault list.
@@ -365,23 +379,30 @@ int xe_pm_runtime_suspend(struct xe_device *xe)
 	if (xe->d3cold.allowed) {
 		err = xe_bo_evict_all(xe);
 		if (err)
-			goto out;
+			goto out_pxp;
 		xe_display_pm_suspend(xe, true);
 	}
 
 	for_each_gt(gt, xe, id) {
 		err = xe_gt_suspend(gt);
 		if (err)
-			goto out;
+			goto out_display;
 	}
 
 	xe_irq_suspend(xe);
 
 	if (xe->d3cold.allowed)
 		xe_display_pm_suspend_late(xe);
+
+	lock_map_release(&xe_pm_runtime_lockdep_map);
+	xe_pm_write_callback_task(xe, NULL);
+	return 0;
+
+out_display:
+	xe_display_pm_resume(xe, true);
+out_pxp:
+	xe_pxp_pm_resume(xe->pxp);
 out:
-	if (err)
-		xe_display_pm_resume(xe, true);
 	lock_map_release(&xe_pm_runtime_lockdep_map);
 	xe_pm_write_callback_task(xe, NULL);
 	return err;
@@ -431,6 +452,9 @@ int xe_pm_runtime_resume(struct xe_device *xe)
 		if (err)
 			goto out;
 	}
+
+	xe_pxp_pm_resume(xe->pxp);
+
 out:
 	lock_map_release(&xe_pm_runtime_lockdep_map);
 	xe_pm_write_callback_task(xe, NULL);
diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
index 493b8ff4b743..a5e912b1fdc1 100644
--- a/drivers/gpu/drm/xe/xe_pxp.c
+++ b/drivers/gpu/drm/xe/xe_pxp.c
@@ -137,6 +137,13 @@ static void pxp_terminate(struct xe_pxp *pxp)
 	if (pxp->status == XE_PXP_ACTIVE)
 		pxp->key_instance++;
 
+	/*
+	 * we'll mark the status as needing termination on resume, so no need to
+	 * emit a termination now.
+	 */
+	if (pxp->status == XE_PXP_SUSPENDED)
+		return;
+
 	/*
 	 * If we have a termination already in progress, we need to wait for
 	 * it to complete before queueing another one. We update the state
@@ -183,17 +190,19 @@ static void pxp_terminate(struct xe_pxp *pxp)
 static void pxp_terminate_complete(struct xe_pxp *pxp)
 {
 	/*
-	 * We expect PXP to be in one of 2 states when we get here:
+	 * We expect PXP to be in one of 3 states when we get here:
 	 * - XE_PXP_TERMINATION_IN_PROGRESS: a single termination event was
 	 * requested and it is now completing, so we're ready to start.
 	 * - XE_PXP_NEEDS_TERMINATION: a second termination was requested while
 	 * the first one was still being processed; we don't update the state
 	 * in this case so the pxp_start code will automatically issue that
 	 * second termination.
+	 * - XE_PXP_SUSPENDED: PXP is now suspended, so we defer everything to
+	 * when we come back on resume.
 	 */
 	if (pxp->status == XE_PXP_TERMINATION_IN_PROGRESS)
 		pxp->status = XE_PXP_READY_TO_START;
-	else if (pxp->status != XE_PXP_NEEDS_TERMINATION)
+	else if (pxp->status != XE_PXP_NEEDS_TERMINATION && pxp->status != XE_PXP_SUSPENDED)
 		drm_err(&pxp->xe->drm,
 			"PXP termination complete while status was %u\n",
 			pxp->status);
@@ -484,6 +493,7 @@ int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type)
 		pxp_terminate(pxp);
 		mutex_unlock(&pxp->mutex);
 		goto wait_for_termination;
+	case XE_PXP_SUSPENDED:
 	default:
 		drm_err(&pxp->xe->drm, "unexpected state during PXP start: %u", pxp->status);
 		ret = -EIO;
@@ -620,3 +630,84 @@ int xe_pxp_key_check(struct xe_pxp *pxp, struct xe_bo *bo)
 
 	return 0;
 }
+
+int xe_pxp_pm_suspend(struct xe_pxp *pxp)
+{
+	int ret = 0;
+
+	if (!xe_pxp_is_enabled(pxp))
+		 return 0;
+
+	mutex_lock(&pxp->mutex);
+
+	/* if the termination is already in progress, no need to re-emit it */
+	if (!completion_done(&pxp->termination))
+		goto mark_suspended;
+
+	switch (pxp->status) {
+	case XE_PXP_ERROR:
+	case XE_PXP_READY_TO_START:
+	case XE_PXP_SUSPENDED:
+		/* nothing to cleanup */
+		break;
+	case XE_PXP_NEEDS_TERMINATION:
+		/* If PXP was never used we can skip the cleanup */
+		if (pxp->key_instance == pxp->last_suspend_key_instance)
+			break;
+		fallthrough;
+	case XE_PXP_ACTIVE:
+		pxp_terminate(pxp);
+		break;
+	default:
+		drm_err(&pxp->xe->drm, "unexpected state during PXP suspend: %u",
+			pxp->status);
+		ret = -EIO;
+		goto out;
+	}
+
+mark_suspended:
+	/*
+	 * We set this even if we were in error state, hoping the suspend clears
+	 * the error. Worse case we fail again and go in error state again.
+	 */
+	pxp->status = XE_PXP_SUSPENDED;
+
+	mutex_unlock(&pxp->mutex);
+
+	/*
+	 * if there is a termination in progress, wait for it.
+	 * We need to wait outside the lock because the completion is done from
+	 * within the lock
+	 */
+	if (!wait_for_completion_timeout(&pxp->termination,
+					 msecs_to_jiffies(PXP_TERMINATION_TIMEOUT_MS)))
+		ret = -ETIMEDOUT;
+
+	pxp->last_suspend_key_instance = pxp->key_instance;
+
+out:
+	return ret;
+}
+
+void xe_pxp_pm_resume(struct xe_pxp *pxp)
+{
+	int err;
+
+	if (!xe_pxp_is_enabled(pxp))
+		 return;
+
+	err = kcr_pxp_enable(pxp);
+
+	mutex_lock(&pxp->mutex);
+
+	xe_assert(pxp->xe, pxp->status == XE_PXP_SUSPENDED);
+
+	if (err)
+		pxp->status = XE_PXP_ERROR;
+	else
+		pxp->status = XE_PXP_NEEDS_TERMINATION;
+
+	mutex_unlock(&pxp->mutex);
+
+	return;
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
index e3c4c92fc7a3..0ff79f9a900b 100644
--- a/drivers/gpu/drm/xe/xe_pxp.h
+++ b/drivers/gpu/drm/xe/xe_pxp.h
@@ -20,6 +20,9 @@ int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
 int xe_pxp_init(struct xe_device *xe);
 void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
 
+int xe_pxp_pm_suspend(struct xe_pxp *pxp);
+void xe_pxp_pm_resume(struct xe_pxp *pxp);
+
 int xe_pxp_exec_queue_add(struct xe_pxp *pxp, struct xe_exec_queue *q, u8 type);
 void xe_pxp_exec_queue_remove(struct xe_pxp *pxp, struct xe_exec_queue *q);
 
diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
index e28bcbda1ec6..58edd0d85348 100644
--- a/drivers/gpu/drm/xe/xe_pxp_types.h
+++ b/drivers/gpu/drm/xe/xe_pxp_types.h
@@ -23,7 +23,8 @@ enum xe_pxp_status {
 	XE_PXP_NEEDS_TERMINATION = 0, /* starting status */
 	XE_PXP_TERMINATION_IN_PROGRESS,
 	XE_PXP_READY_TO_START,
-	XE_PXP_ACTIVE
+	XE_PXP_ACTIVE,
+	XE_PXP_SUSPENDED
 };
 
 /**
@@ -91,6 +92,12 @@ struct xe_pxp {
 
 	/** @key_instance: keep track of the current iteration of the PXP key */
 	u32 key_instance;
+	/**
+	 * @last_suspend_key_instance: value of key_instance at the last
+	 * suspend. Used to check if any PXP session has been created between
+	 * suspend cycles.
+	 */
+	u32 last_suspend_key_instance;
 };
 
 #endif /* __XE_PXP_TYPES_H__ */
-- 
2.43.0

[RFC 13/14] drm/xe/pxp: Add PXP debugfs support

[Daniele Ceraolo Spurio]

This patch introduces 2 PXP debugfs entries:

- info: prints the current PXP status and key instance
- terminate: simulate a termination interrupt

The first one is useful for debug, while the second one can be used for
testing the termination flow.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/Makefile         |   1 +
 drivers/gpu/drm/xe/xe_debugfs.c     |   3 +
 drivers/gpu/drm/xe/xe_pxp_debugfs.c | 120 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pxp_debugfs.h |  13 +++
 4 files changed, 137 insertions(+)
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.c
 create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.h

diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
index a4514265085b..688decad4c75 100644
--- a/drivers/gpu/drm/xe/Makefile
+++ b/drivers/gpu/drm/xe/Makefile
@@ -105,6 +105,7 @@ xe-y += xe_bb.o \
 	xe_pt.o \
 	xe_pt_walk.o \
 	xe_pxp.o \
+	xe_pxp_debugfs.o \
 	xe_pxp_submit.o \
 	xe_query.o \
 	xe_range_fence.o \
diff --git a/drivers/gpu/drm/xe/xe_debugfs.c b/drivers/gpu/drm/xe/xe_debugfs.c
index 1011e5d281fa..a04f9c2d886b 100644
--- a/drivers/gpu/drm/xe/xe_debugfs.c
+++ b/drivers/gpu/drm/xe/xe_debugfs.c
@@ -17,6 +17,7 @@
 #include "xe_gt_printk.h"
 #include "xe_guc_ads.h"
 #include "xe_pm.h"
+#include "xe_pxp_debugfs.h"
 #include "xe_sriov.h"
 #include "xe_step.h"
 
@@ -214,6 +215,8 @@ void xe_debugfs_register(struct xe_device *xe)
 	for_each_gt(gt, xe, id)
 		xe_gt_debugfs_register(gt);
 
+	xe_pxp_debugfs_register(xe->pxp);
+
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("fail_gt_reset", root, &gt_reset_failure);
 #endif
diff --git a/drivers/gpu/drm/xe/xe_pxp_debugfs.c b/drivers/gpu/drm/xe/xe_pxp_debugfs.c
new file mode 100644
index 000000000000..00c8179a9f0f
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp_debugfs.c
@@ -0,0 +1,120 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#include "xe_pxp_debugfs.h"
+
+#include <linux/debugfs.h>
+
+#include <drm/drm_debugfs.h>
+#include <drm/drm_managed.h>
+#include <drm/drm_print.h>
+
+#include "xe_device.h"
+#include "xe_pxp.h"
+#include "xe_pxp_types.h"
+#include "regs/xe_gt_regs.h"
+
+static struct xe_pxp *node_to_pxp(struct drm_info_node *node)
+{
+	return node->info_ent->data;
+}
+
+static const char *pxp_status_to_str(struct xe_pxp *pxp)
+{
+	lockdep_assert_held(&pxp->mutex);
+
+	switch (pxp->status) {
+	case XE_PXP_ERROR:
+		return "error";
+	case XE_PXP_NEEDS_TERMINATION:
+		return "needs termination";
+	case XE_PXP_TERMINATION_IN_PROGRESS:
+		return "termination in progress";
+	case XE_PXP_READY_TO_START:
+		return "ready to start";
+	case XE_PXP_ACTIVE:
+		return "active";
+	case XE_PXP_SUSPENDED:
+		return "suspended";
+	default:
+		return "unknown";
+	}
+};
+
+static int pxp_info(struct seq_file *m, void *data)
+{
+	struct xe_pxp *pxp = node_to_pxp(m->private);
+	struct drm_printer p = drm_seq_file_printer(m);
+	const char *status;
+
+	if (!xe_pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	mutex_lock(&pxp->mutex);
+	status = pxp_status_to_str(pxp);
+	mutex_unlock(&pxp->mutex);
+
+	drm_printf(&p, "status: %s\n", status);
+	drm_printf(&p, "instance counter: %u\n", pxp->key_instance);
+
+	return 0;
+}
+
+static int pxp_terminate(struct seq_file *m, void *data)
+{
+	struct xe_pxp *pxp = node_to_pxp(m->private);
+	struct drm_printer p = drm_seq_file_printer(m);
+
+	if (!xe_pxp_is_enabled(pxp))
+		return -ENODEV;
+
+	/* simulate a termination interrupt */
+	spin_lock_irq(&pxp->xe->irq.lock);
+	xe_pxp_irq_handler(pxp->xe, KCR_PXP_STATE_TERMINATED_INTERRUPT);
+	spin_unlock_irq(&pxp->xe->irq.lock);
+
+	drm_printf(&p, "Termination queued\n");
+
+	return 0;
+}
+
+static const struct drm_info_list debugfs_list[] = {
+	{"info", pxp_info, 0},
+	{"terminate", pxp_terminate, 0},
+};
+
+void xe_pxp_debugfs_register(struct xe_pxp *pxp)
+{
+	struct drm_minor *minor;
+	struct drm_info_list *local;
+	struct dentry *root;
+	int i;
+
+	if (!xe_pxp_is_enabled(pxp))
+		return;
+
+	minor = pxp->xe->drm.primary;
+	if (!minor->debugfs_root)
+		return;
+
+#define DEBUGFS_SIZE	(ARRAY_SIZE(debugfs_list) * sizeof(struct drm_info_list))
+	local = drmm_kmalloc(&pxp->xe->drm, DEBUGFS_SIZE, GFP_KERNEL);
+	if (!local)
+		return;
+
+	memcpy(local, debugfs_list, DEBUGFS_SIZE);
+#undef DEBUGFS_SIZE
+
+	for (i = 0; i < ARRAY_SIZE(debugfs_list); ++i)
+		local[i].data = pxp;
+
+	root = debugfs_create_dir("pxp", minor->debugfs_root);
+	if (IS_ERR(root))
+		return;
+
+	drm_debugfs_create_files(local,
+				 ARRAY_SIZE(debugfs_list),
+				 root, minor);
+}
diff --git a/drivers/gpu/drm/xe/xe_pxp_debugfs.h b/drivers/gpu/drm/xe/xe_pxp_debugfs.h
new file mode 100644
index 000000000000..988466aad50b
--- /dev/null
+++ b/drivers/gpu/drm/xe/xe_pxp_debugfs.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __XE_PXP_DEBUGFS_H__
+#define __XE_PXP_DEBUGFS_H__
+
+struct xe_pxp;
+
+void xe_pxp_debugfs_register(struct xe_pxp *pxp);
+
+#endif /* __XE_PXP_DEBUGFS_H__ */
-- 
2.43.0

[RFC 14/14] drm/xe/pxp: Enable PXP for MTL

[Daniele Ceraolo Spurio]

Now that are the pieces are there, we can turn the feature on.

Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/xe/xe_pci.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c
index de0d9f8bba2c..764f58003bea 100644
--- a/drivers/gpu/drm/xe/xe_pci.c
+++ b/drivers/gpu/drm/xe/xe_pci.c
@@ -333,6 +333,7 @@ static const struct xe_device_desc mtl_desc = {
 	.require_force_probe = true,
 	PLATFORM(METEORLAKE),
 	.has_display = true,
+	.has_pxp = true,
 };
 
 static const struct xe_device_desc lnl_desc = {
-- 
2.43.0

✗ CI.Patch_applied: failure for Add PXP HWDRM support

[Patchwork]

== Series Details ==

Series: Add PXP HWDRM support
URL   : https://patchwork.freedesktop.org/series/136052/
State : failure

== Summary ==

=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: f860f222eabb drm-tip: 2024y-07m-12d-21h-27m-34s UTC integration manifest
=== git am output follows ===
error: patch failed: drivers/gpu/drm/xe/xe_device.c:46
error: drivers/gpu/drm/xe/xe_device.c: patch does not apply
error: patch failed: drivers/gpu/drm/xe/xe_device_types.h:466
error: drivers/gpu/drm/xe/xe_device_types.h: patch does not apply
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Applying: drm/xe: Make exec_queue_kill safe to call twice
Applying: drm/xe/pxp: Initialize PXP structure and KCR reg
Patch failed at 0002 drm/xe/pxp: Initialize PXP structure and KCR reg
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Re: [RFC 01/14] drm/xe: Make exec_queue_kill safe to call twice

[Matthew Brost]

On Fri, Jul 12, 2024 at 02:28:45PM -0700, Daniele Ceraolo Spurio wrote:
> An upcoming PXP patch will kill queues at runtime when a PXP
> invalidation event occurs, so we need exec_queue_kill to be safe to call
> multiple times.
> 
> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> Cc: Matthew Brost <matthew.brost@intel.com>

Not opposed this patch but is this really required for PXP as PXP queue
are not in LR mode are they?

Either way, I'm open to accepting this as xe_exec_queue_kill probably
should be safe to call twice anyways.

Matt

> ---
>  drivers/gpu/drm/xe/xe_vm.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index 5b166fa03684..02f684c0330d 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c
> @@ -280,8 +280,10 @@ void xe_vm_remove_compute_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
>  		return;
>  
>  	down_write(&vm->lock);
> -	list_del(&q->lr.link);
> -	--vm->preempt.num_exec_queues;
> +	if (!list_empty(&q->lr.link)) {
> +		list_del_init(&q->lr.link);
> +		--vm->preempt.num_exec_queues;
> +	}
>  	if (q->lr.pfence) {
>  		dma_fence_enable_sw_signaling(q->lr.pfence);
>  		dma_fence_put(q->lr.pfence);
> -- 
> 2.43.0
> 

Re: [RFC 01/14] drm/xe: Make exec_queue_kill safe to call twice

[Daniele Ceraolo Spurio]

On 7/12/2024 3:11 PM, Matthew Brost wrote:
> On Fri, Jul 12, 2024 at 02:28:45PM -0700, Daniele Ceraolo Spurio wrote:
>> An upcoming PXP patch will kill queues at runtime when a PXP
>> invalidation event occurs, so we need exec_queue_kill to be safe to call
>> multiple times.
>>
>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>> Cc: Matthew Brost <matthew.brost@intel.com>
> Not opposed this patch but is this really required for PXP as PXP queue
> are not in LR mode are they?
>
> Either way, I'm open to accepting this as xe_exec_queue_kill probably
> should be safe to call twice anyways.

Not required for PXP, no. I just really didn't like the idea of calling 
a function twice when it wasn't designed to support multiple calls, even 
if it didn't apply to my specific case.

Daniele

>
> Matt
>
>> ---
>>   drivers/gpu/drm/xe/xe_vm.c | 6 ++++--
>>   1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
>> index 5b166fa03684..02f684c0330d 100644
>> --- a/drivers/gpu/drm/xe/xe_vm.c
>> +++ b/drivers/gpu/drm/xe/xe_vm.c
>> @@ -280,8 +280,10 @@ void xe_vm_remove_compute_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
>>   		return;
>>   
>>   	down_write(&vm->lock);
>> -	list_del(&q->lr.link);
>> -	--vm->preempt.num_exec_queues;
>> +	if (!list_empty(&q->lr.link)) {
>> +		list_del_init(&q->lr.link);
>> +		--vm->preempt.num_exec_queues;
>> +	}
>>   	if (q->lr.pfence) {
>>   		dma_fence_enable_sw_signaling(q->lr.pfence);
>>   		dma_fence_put(q->lr.pfence);
>> -- 
>> 2.43.0
>>

Re: [RFC 03/14] drm/xe/pxp: Allocate PXP execution resources

[Matthew Brost]

On Fri, Jul 12, 2024 at 02:28:47PM -0700, Daniele Ceraolo Spurio wrote:
> PXP requires submissions to the HW for the following operations
> 
> 1) Key invalidation, done via the VCS engine
> 2) Communication with the GSC FW for session management, done via the
>    GSCCS
> 
> For #1 we can allocate a simple kernel context, but #2 requires the
> submissions to be done with PPGTT, which is not currently supported in Xe.
> To add this support, the following changes have been included:
> 
> - a new type of kernel-owned VM (marked as GSC)
> - a new function to map a BO into a VM from within the kernel
> 
> RFC note: I've tweaked some of the VM functions to return the fence
> further up the stack, so I can wait on it from the PXP code. Not sure if
> this is the best approach.
> 
> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> Cc: Matthew Brost <matthew.brost@intel.com>

Not a complete review but adding some thoughts. Looks sane enough to me.

Random musing and nits below.

> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
>  drivers/gpu/drm/xe/Makefile                   |   1 +
>  drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |   7 +
>  drivers/gpu/drm/xe/xe_exec_queue.c            |   3 +
>  drivers/gpu/drm/xe/xe_pxp.c                   |  25 ++-
>  drivers/gpu/drm/xe/xe_pxp_submit.c            | 188 ++++++++++++++++++
>  drivers/gpu/drm/xe/xe_pxp_submit.h            |  16 ++
>  drivers/gpu/drm/xe/xe_pxp_types.h             |  33 +++
>  drivers/gpu/drm/xe/xe_vm.c                    | 100 +++++++++-
>  drivers/gpu/drm/xe/xe_vm.h                    |   6 +
>  drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
>  10 files changed, 372 insertions(+), 8 deletions(-)
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h
> 
> diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
> index 5f15e6dd5057..a4514265085b 100644
> --- a/drivers/gpu/drm/xe/Makefile
> +++ b/drivers/gpu/drm/xe/Makefile
> @@ -105,6 +105,7 @@ xe-y += xe_bb.o \
>  	xe_pt.o \
>  	xe_pt_walk.o \
>  	xe_pxp.o \
> +	xe_pxp_submit.o \
>  	xe_query.o \
>  	xe_range_fence.o \
>  	xe_reg_sr.o \
> diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> index 57520809e48d..f3c4cf10ba20 100644
> --- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> +++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> @@ -6,6 +6,7 @@
>  #ifndef _ABI_GSC_PXP_COMMANDS_ABI_H
>  #define _ABI_GSC_PXP_COMMANDS_ABI_H
>  
> +#include <linux/sizes.h>
>  #include <linux/types.h>
>  
>  /* Heci client ID for PXP commands */
> @@ -13,6 +14,12 @@
>  
>  #define PXP_APIVER(x, y) (((x) & 0xFFFF) << 16 | ((y) & 0xFFFF))
>  
> +/*
> + * A PXP sub-section in an HECI packet can be up to 64K big in each direction.
> + * This does not include the top-level GSC header.
> + */
> +#define PXP_MAX_PACKET_SIZE SZ_64K
> +
>  /*
>   * there are a lot of status codes for PXP, but we only define the cross-API
>   * common ones that we actually can handle in the kernel driver. Other failure
> diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
> index 0ba37835849b..bc6e867aba17 100644
> --- a/drivers/gpu/drm/xe/xe_exec_queue.c
> +++ b/drivers/gpu/drm/xe/xe_exec_queue.c
> @@ -131,6 +131,9 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
>  	struct xe_exec_queue *q;
>  	int err;
>  
> +	/* VMs for GSCCS queues (and only those) must have the XE_VM_FLAG_GSC flag */
> +	xe_assert(xe, !vm || (!!(vm->flags & XE_VM_FLAG_GSC) == !!(hwe->engine_id == XE_HW_ENGINE_GSCCS0)));
> +

We should be able to remove this soon. More on that below.

>  	q = __xe_exec_queue_alloc(xe, vm, logical_mask, width, hwe, flags,
>  				  extensions);
>  	if (IS_ERR(q))
> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> index cdb29b104006..01386b9f0c50 100644
> --- a/drivers/gpu/drm/xe/xe_pxp.c
> +++ b/drivers/gpu/drm/xe/xe_pxp.c
> @@ -12,6 +12,7 @@
>  #include "xe_gt.h"
>  #include "xe_gt_types.h"
>  #include "xe_mmio.h"
> +#include "xe_pxp_submit.h"
>  #include "xe_pxp_types.h"
>  #include "xe_uc_fw.h"
>  #include "regs/xe_pxp_regs.h"
> @@ -50,6 +51,20 @@ static int kcr_pxp_enable(const struct xe_pxp *pxp)
>  	return kcr_pxp_set_status(pxp, true);
>  }
>  
> +static int kcr_pxp_disable(const struct xe_pxp *pxp)
> +{
> +	return kcr_pxp_set_status(pxp, false);
> +}
> +
> +static void pxp_fini(void *arg)
> +{
> +	struct xe_pxp *pxp = arg;
> +
> +	xe_pxp_destroy_execution_resources(pxp);
> +
> +	/* no need to explicitly disable KCR since we're going to do an FLR */
> +}
> +
>  /**
>   * xe_pxp_init - initialize PXP support
>   * @xe: the xe_device structure
> @@ -97,7 +112,15 @@ int xe_pxp_init(struct xe_device *xe)
>  	if (err)
>  		return err;
>  
> +	err = xe_pxp_allocate_execution_resources(pxp);
> +	if (err)
> +		goto kcr_disable;
> +
>  	xe->pxp = pxp;
>  
> -	return 0;
> +	return devm_add_action_or_reset(xe->drm.dev, pxp_fini, pxp);
> +
> +kcr_disable:
> +	kcr_pxp_disable(pxp);
> +	return err;
>  }
> diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
> new file mode 100644
> index 000000000000..4fc3c7c58101
> --- /dev/null
> +++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
> @@ -0,0 +1,188 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright(c) 2024 Intel Corporation.
> + */
> +
> +#include "xe_pxp_submit.h"
> +
> +#include <drm/xe_drm.h>
> +
> +#include "xe_device_types.h"
> +#include "xe_bo.h"
> +#include "xe_exec_queue.h"
> +#include "xe_gsc_submit.h"
> +#include "xe_gt.h"
> +#include "xe_pxp_types.h"
> +#include "xe_vm.h"
> +#include "regs/xe_gt_regs.h"
> +
> +static int create_vcs_context(struct xe_pxp *pxp)
> +{
> +	struct xe_gt *gt = pxp->gt;
> +	struct xe_hw_engine *hwe;
> +	struct xe_exec_queue *q;
> +
> +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_VIDEO_DECODE, 0, true);
> +	if (!hwe)
> +		return -ENODEV;
> +

Ugh, really want to completely decouple an exec queue from hwe (e.g.
don't pass in hwe to xe_exec_queue_create). I guess this already in code
so fine here just a reminder of this ugliness.

> +	q = xe_exec_queue_create(pxp->xe, NULL, BIT(hwe->logical_instance), 1, hwe,
> +				 EXEC_QUEUE_FLAG_KERNEL | EXEC_QUEUE_FLAG_PERMANENT, 0);
> +	if (IS_ERR(q))
> +		return PTR_ERR(q);
> +
> +	pxp->vcs_queue = q;
> +

So how is this used? Not attached to a VM? GGTT or ring instructions
only? Any downside of attaching this to GSC VM?

> +	return 0;
> +}
> +
> +static void destroy_vcs_context(struct xe_pxp *pxp)
> +{
> +	if (pxp->vcs_queue)
> +		xe_exec_queue_put(pxp->vcs_queue);
> +}
> +
> +/*
> + * We allocate a single object for the batch and the input and output BOs. PXP
> + * commands can require a lot of BO space (see PXP_MAX_PACKET_SIZE), but we
> + * currently only support a subset of commands that are small (< 20 dwords),
> + * so a single page is enough for now.
> + */
> +#define PXP_BB_SIZE		XE_PAGE_SIZE
> +#define PXP_INOUT_SIZE		XE_PAGE_SIZE
> +#define PXP_BO_SIZE		(PXP_BB_SIZE + (2 * PXP_INOUT_SIZE))
> +#define PXP_BB_OFFSET		0
> +#define PXP_MSG_IN_OFFSET 	PXP_BB_SIZE
> +#define PXP_MSG_OUT_OFFSET 	(PXP_MSG_IN_OFFSET + PXP_INOUT_SIZE)
> +static int allocate_gsc_execution_resources(struct xe_pxp *pxp)
> +{
> +	struct xe_gt *gt = pxp->gt;
> +	struct xe_tile *tile = gt_to_tile(gt);
> +	struct xe_device *xe = pxp->xe;
> +	struct xe_hw_engine *hwe;
> +	struct xe_vm *vm;
> +	struct xe_bo *bo;
> +	struct xe_exec_queue *q;
> +	struct dma_fence *fence;
> +	long timeout;
> +	int err = 0;
> +
> +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_OTHER, OTHER_GSC_INSTANCE, false);
> +
> +	/* we shouldn't reach here if the GSC engine is not available */
> +	xe_assert(xe, hwe);
> +
> +	/* PXP instructions must be issued from PPGTT */
> +	vm = xe_vm_create(xe, XE_VM_FLAG_GSC);
> +	if (IS_ERR(vm))
> +		return PTR_ERR(vm);
> +
> +	/* We allocate a single object for the batch and the in/out memory */
> +	xe_vm_lock(vm, false);
> +	bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BO_SIZE, ttm_bo_type_kernel,
> +				  XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_NEEDS_UC);
> +	xe_vm_unlock(vm);
> +	if (IS_ERR(bo)) {
> +		err = PTR_ERR(bo);
> +		goto vm_out;
> +	}
> +
> +	fence = xe_vm_bind_bo(vm, bo, NULL, 0, XE_CACHE_WB);
> +	if (IS_ERR(fence)) {
> +		err = PTR_ERR(fence);
> +		goto bo_out;
> +	}
> +
> +	timeout = dma_fence_wait_timeout(fence, false, HZ);
> +	dma_fence_put(fence);
> +	if (timeout <= 0) {
> +		err = timeout ?: -ETIME;
> +		goto bo_out;
> +	}
> +
> +	q = xe_exec_queue_create(xe, vm, BIT(hwe->logical_instance), 1, hwe,
> +				 EXEC_QUEUE_FLAG_KERNEL |
> +				 EXEC_QUEUE_FLAG_PERMANENT, 0);
> +	if (IS_ERR(q)) {
> +		err = PTR_ERR(q);
> +		goto bo_out;
> +	}
> +
> +	pxp->gsc_exec.vm = vm;
> +	pxp->gsc_exec.bo = bo;
> +	pxp->gsc_exec.batch = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_BB_OFFSET);
> +	pxp->gsc_exec.msg_in = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_IN_OFFSET);
> +	pxp->gsc_exec.msg_out = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_OUT_OFFSET);

So with this mapping, all GSC are serially executed and waited on. There
won't ever be a need to pipeline things? If the later is true you could
xe_bb_* plus suballocation of the BO you map. More complex so if serial
execute is all you will ever need, then yea probably don't use that.

> +	pxp->gsc_exec.q = q;
> +
> +	/* initialize host-session-handle (for all Xe-to-gsc-firmware PXP cmds) */
> +	pxp->gsc_exec.host_session_handle = xe_gsc_create_host_session_id();
> +
> +	return 0;
> +
> +bo_out:
> +	xe_vm_lock(vm, false);
> +	xe_bo_unpin(bo);
> +	xe_vm_unlock(vm);
> +
> +	xe_bo_put(bo);

Can use helper I mention below.

> +vm_out:
> +	xe_vm_close_and_put(vm);
> +
> +	return err;
> +}
> +
> +static void destroy_gsc_execution_resources(struct xe_pxp *pxp)
> +{
> +	if (!pxp->gsc_exec.q)
> +		return;
> +
> +	iosys_map_clear(&pxp->gsc_exec.msg_out);
> +	iosys_map_clear(&pxp->gsc_exec.msg_in);
> +	iosys_map_clear(&pxp->gsc_exec.batch);

I don't think this is strickly need as it just sets a pointer to NULL.

> +
> +	xe_exec_queue_put(pxp->gsc_exec.q);
> +
> +	xe_vm_lock(pxp->gsc_exec.vm, false);
> +	xe_bo_unpin(pxp->gsc_exec.bo);
> +	xe_vm_unlock(pxp->gsc_exec.vm);
> +	xe_bo_put(pxp->gsc_exec.bo);
> +

This looks awfully like xe_bo_unpin_map_no_vm. Maybe rename that
function and just use it?

If a BO is private to a VM (this one is, xe_bo_lock and xe_vm_lock mean
the same thing).

> +	xe_vm_close_and_put(pxp->gsc_exec.vm);
> +}
> +
> +/**
> + * xe_pxp_allocate_execution_resources - Allocate PXP submission objects
> + * @pxp: the xe_pxp structure
> + *
> + * Allocates exec_queues objects for VCS and GSCCS submission. The GSCCS
> + * submissions are done via PPGTT, so this function allocates a VM for it and
> + * maps the object into it.
> + *
> + * Returns 0 if the allocation and mapping is successful, an errno value
> + * otherwise.
> + */
> +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp)
> +{
> +	int err;
> +
> +	err = create_vcs_context(pxp);
> +	if (err)
> +		return err;
> +
> +	err = allocate_gsc_execution_resources(pxp);
> +	if (err)
> +		goto destroy_vcs_context;
> +
> +	return 0;
> +
> +destroy_vcs_context:
> +	destroy_vcs_context(pxp);
> +	return err;
> +}
> +
> +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp)
> +{
> +	destroy_gsc_execution_resources(pxp);
> +	destroy_vcs_context(pxp);
> +}
> diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
> new file mode 100644
> index 000000000000..1a971fadc081
> --- /dev/null
> +++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
> @@ -0,0 +1,16 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright(c) 2024, Intel Corporation. All rights reserved.
> + */
> +
> +#ifndef __XE_PXP_SUBMIT_H__
> +#define __XE_PXP_SUBMIT_H__
> +
> +#include <linux/types.h>
> +
> +struct xe_pxp;
> +
> +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
> +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
> +
> +#endif /* __XE_PXP_SUBMIT_H__ */
> diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
> index 1561e3bd2676..c16813253b47 100644
> --- a/drivers/gpu/drm/xe/xe_pxp_types.h
> +++ b/drivers/gpu/drm/xe/xe_pxp_types.h
> @@ -6,10 +6,14 @@
>  #ifndef __XE_PXP_TYPES_H__
>  #define __XE_PXP_TYPES_H__
>  
> +#include <linux/iosys-map.h>
>  #include <linux/types.h>
>  
> +struct xe_bo;
> +struct xe_exec_queue;
>  struct xe_device;
>  struct xe_gt;
> +struct xe_vm;
>  
>  /**
>   * struct xe_pxp - pxp state
> @@ -23,6 +27,35 @@ struct xe_pxp {
>  	 * (VDBOX, KCR and GSC)
>  	 */
>  	struct xe_gt *gt;
> +
> +	/** @vcs_queue: kernel-owned VCS exec queue used for PXP operations */
> +	struct xe_exec_queue *vcs_queue;
> +
> +	/** @gsc_exec: kernel-owned objects for PXP submissions to the GSCCS */
> +	struct {
> +		/**
> +		 * @gsc_exec.host_session_handle: handle used in communications
> +		 * with the GSC firmware.
> +		 */
> +		u64 host_session_handle;
> +		/** @gsc_exec.vm: VM used for PXP submissions to the GSCCS */
> +		struct xe_vm *vm;
> +		/** @gsc_exec.q: GSCCS exec queue for PXP submissions */
> +		struct xe_exec_queue *q;
> +
> +		/**
> +		 * @gsc_exec.bo: BO used for submissions to the GSCCS and GSC
> +		 * FW. It includes space for the GSCCS batch and the
> +		 * input/output buffers read/written by the FW
> +		 */
> +		struct xe_bo *bo;
> +		/** @gsc_exec.batch: iosys_map to the batch memory within the BO */
> +		struct iosys_map batch;
> +		/** @gsc_exec.msg_in: iosys_map to the input memory within the BO */
> +		struct iosys_map msg_in;
> +		/** @gsc_exec.msg_out: iosys_map to the output memory within the BO */
> +		struct iosys_map msg_out;
> +	} gsc_exec;
>  };
>  
>  #endif /* __XE_PXP_TYPES_H__ */
> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index 02f684c0330d..412ec9cb9650 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c
> @@ -1315,6 +1315,15 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
>  	struct xe_tile *tile;
>  	u8 id;
>  
> +	/*
> +	 * All GSC VMs are owned by the kernel and can also only be used on
> +	 * the GSCCS. We don't want a kernel-owned VM to put the device in
> +	 * either fault or not fault mode, so we need to exclude the GSC VMs
> +	 * from that count; this is only safe if we ensure that all GSC VMs are
> +	 * non-faulting.
> +	 */
> +	xe_assert(xe, !((flags & XE_VM_FLAG_GSC) && (flags & XE_VM_FLAG_FAULT_MODE)));
> +
>  	vm = kzalloc(sizeof(*vm), GFP_KERNEL);
>  	if (!vm)
>  		return ERR_PTR(-ENOMEM);
> @@ -1442,7 +1451,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
>  	mutex_lock(&xe->usm.lock);
>  	if (flags & XE_VM_FLAG_FAULT_MODE)
>  		xe->usm.num_vm_in_fault_mode++;
> -	else if (!(flags & XE_VM_FLAG_MIGRATION))
> +	else if (!(flags & (XE_VM_FLAG_MIGRATION | XE_VM_FLAG_GSC)))

This change is good now but should become unnecessary once Francois
lands some code to remove the restriction of mixing faulting and
non-faulting VM within a device.

>  		xe->usm.num_vm_in_non_fault_mode++;
>  	mutex_unlock(&xe->usm.lock);
>  
> @@ -2867,11 +2876,10 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
>  	for (i = 0; i < vops->num_syncs; i++)
>  		xe_sync_entry_signal(vops->syncs + i, fence);
>  	xe_exec_queue_last_fence_set(wait_exec_queue, vm, fence);
> -	dma_fence_put(fence);

Nit: I'd send this change and associated change in xe_vm_bind_ioctl +
vm_bind_ioctl_ops_execute in its own patch, perhaps even as an
independent series which I'd RB immediately.

Change looks good though and could be useful else where too.

>  }
>  
> -static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> -				     struct xe_vma_ops *vops)
> +static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> +						   struct xe_vma_ops *vops)
>  {
>  	struct drm_exec exec;
>  	struct dma_fence *fence;
> @@ -2889,7 +2897,6 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>  
>  		fence = ops_execute(vm, vops);
>  		if (IS_ERR(fence)) {
> -			err = PTR_ERR(fence);
>  			/* FIXME: Killing VM rather than proper error handling */
>  			xe_vm_kill(vm, false);

Looks like you are on old baseline before this series landed [1]. I
suggest rebasing as those changes creep up in the upper layers a bit.

[1] https://patchwork.freedesktop.org/series/133034/

>  			goto unlock;
> @@ -2900,7 +2907,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>  
>  unlock:
>  	drm_exec_fini(&exec);
> -	return err;
> +	return fence;
>  }
>  
>  #define SUPPORTED_FLAGS	\
> @@ -3114,6 +3121,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>  	struct xe_sync_entry *syncs = NULL;
>  	struct drm_xe_vm_bind_op *bind_ops;
>  	struct xe_vma_ops vops;
> +	struct dma_fence *fence;
>  	int err;
>  	int i;
>  
> @@ -3264,7 +3272,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>  		goto unwind_ops;
>  	}
>  
> -	err = vm_bind_ioctl_ops_execute(vm, &vops);
> +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
> +	if (IS_ERR(fence))
> +		err = PTR_ERR(fence);
> +	else
> +		dma_fence_put(fence);
>  
>  unwind_ops:
>  	if (err && err != -ENODATA)
> @@ -3297,6 +3309,80 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>  	return err;
>  }
>  
> +/**
> + * xe_vm_bind_bo - bind a kernel BO to a VM
> + * @vm: VM to bind the BO to
> + * @bo: BO to bind
> + * @q: exec queue to use for the bind (optional)
> + * @addr: address at which to bind the BO
> + * @cache_lvl: PAT cache level to use
> + *
> + * Execute a VM bind map operation on a kernel-owned BO to bind it into a
> + * kernel-owned VM.
> + *
> + * Returns 0 if the ops execution is successful, an errno value otherwise.
> + * TODO: return a fence instead.
> + */
> +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
> +				struct xe_exec_queue *q, u64 addr,
> +				enum xe_cache_level cache_lvl)
> +{
> +	struct xe_vma_ops vops;
> +	struct drm_gpuva_ops *ops = NULL;
> +	struct dma_fence *fence;
> +	int err;
> +
> +	xe_bo_get(bo);
> +	xe_vm_get(vm);
> +	if (q)
> +		xe_exec_queue_get(q);
> +
> +	down_write(&vm->lock);
> +
> +	xe_vma_ops_init(&vops, vm, q, NULL, 0);
> +
> +	ops = vm_bind_ioctl_ops_create(vm, bo, 0, addr, bo->size,
> +				       DRM_XE_VM_BIND_OP_MAP, 0,
> +				       vm->xe->pat.idx[cache_lvl], 0);
> +	if (IS_ERR(ops)) {
> +		err = PTR_ERR(ops);
> +		goto release_vm_lock;
> +	}
> +
> +	err = vm_bind_ioctl_ops_parse(vm, q, ops, NULL, 0, &vops, true);
> +	if (err)
> +		goto release_vm_lock;
> +
> +	/* Nothing to do */
> +	if (list_empty(&vops.list)) {

Can this ever be true? In the current usage it appear so. Maybe convert
to an asset !list_empty to simplify this function slightly?

Matt

> +		err = -ENODATA;
> +		goto unwind_ops;
> +	}
> +
> +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
> +	if (IS_ERR(fence))
> +		err = PTR_ERR(fence);
> +
> +unwind_ops:
> +	if (err && err != -ENODATA)
> +		vm_bind_ioctl_ops_unwind(vm, &ops, 1);
> +
> +	drm_gpuva_ops_free(&vm->gpuvm, ops);
> +
> +release_vm_lock:
> +	up_write(&vm->lock);
> +
> +	if (q)
> +		xe_exec_queue_put(q);
> +	xe_vm_put(vm);
> +	xe_bo_put(bo);
> +
> +	if (err)
> +		fence = ERR_PTR(err);
> +
> +	return fence;
> +}
> +
>  /**
>   * xe_vm_lock() - Lock the vm's dma_resv object
>   * @vm: The struct xe_vm whose lock is to be locked
> diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
> index b481608b12f1..5e298ac90dfc 100644
> --- a/drivers/gpu/drm/xe/xe_vm.h
> +++ b/drivers/gpu/drm/xe/xe_vm.h
> @@ -19,6 +19,8 @@ struct drm_file;
>  struct ttm_buffer_object;
>  struct ttm_validate_buffer;
>  
> +struct dma_fence;
> +
>  struct xe_exec_queue;
>  struct xe_file;
>  struct xe_sync_entry;
> @@ -248,6 +250,10 @@ int xe_vm_lock_vma(struct drm_exec *exec, struct xe_vma *vma);
>  int xe_vm_validate_rebind(struct xe_vm *vm, struct drm_exec *exec,
>  			  unsigned int num_fences);
>  
> +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
> +				struct xe_exec_queue *q, u64 addr,
> +				enum xe_cache_level cache_lvl);
> +
>  /**
>   * xe_vm_resv() - Return's the vm's reservation object
>   * @vm: The vm
> diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
> index ce1a63a5e3e7..60ce327d303c 100644
> --- a/drivers/gpu/drm/xe/xe_vm_types.h
> +++ b/drivers/gpu/drm/xe/xe_vm_types.h
> @@ -152,6 +152,7 @@ struct xe_vm {
>  #define XE_VM_FLAG_BANNED		BIT(5)
>  #define XE_VM_FLAG_TILE_ID(flags)	FIELD_GET(GENMASK(7, 6), flags)
>  #define XE_VM_FLAG_SET_TILE_ID(tile)	FIELD_PREP(GENMASK(7, 6), (tile)->id)
> +#define XE_VM_FLAG_GSC			BIT(8)
>  	unsigned long flags;
>  
>  	/** @composite_fence_ctx: context composite fence */
> -- 
> 2.43.0
> 

Re: [RFC 01/14] drm/xe: Make exec_queue_kill safe to call twice

[Matthew Brost]

On Fri, Jul 12, 2024 at 03:22:43PM -0700, Daniele Ceraolo Spurio wrote:
> 
> 
> On 7/12/2024 3:11 PM, Matthew Brost wrote:
> > On Fri, Jul 12, 2024 at 02:28:45PM -0700, Daniele Ceraolo Spurio wrote:
> > > An upcoming PXP patch will kill queues at runtime when a PXP
> > > invalidation event occurs, so we need exec_queue_kill to be safe to call
> > > multiple times.
> > > 
> > > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> > > Cc: Matthew Brost <matthew.brost@intel.com>
> > Not opposed this patch but is this really required for PXP as PXP queue
> > are not in LR mode are they?
> > 
> > Either way, I'm open to accepting this as xe_exec_queue_kill probably
> > should be safe to call twice anyways.
> 
> Not required for PXP, no. I just really didn't like the idea of calling a
> function twice when it wasn't designed to support multiple calls, even if it
> didn't apply to my specific case.
> 

Fair enough, I think go ahead and send this out by itself with:
Reviewed-by: Matthew Brost <matthew.brost@intel.com>

Maybe add some kernel doc to xe_exec_queue_kill too if have time
mentioning it is safe to call twice.  

> Daniele
> 
> > 
> > Matt
> > 
> > > ---
> > >   drivers/gpu/drm/xe/xe_vm.c | 6 ++++--
> > >   1 file changed, 4 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> > > index 5b166fa03684..02f684c0330d 100644
> > > --- a/drivers/gpu/drm/xe/xe_vm.c
> > > +++ b/drivers/gpu/drm/xe/xe_vm.c
> > > @@ -280,8 +280,10 @@ void xe_vm_remove_compute_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
> > >   		return;
> > >   	down_write(&vm->lock);
> > > -	list_del(&q->lr.link);
> > > -	--vm->preempt.num_exec_queues;
> > > +	if (!list_empty(&q->lr.link)) {
> > > +		list_del_init(&q->lr.link);
> > > +		--vm->preempt.num_exec_queues;
> > > +	}
> > >   	if (q->lr.pfence) {
> > >   		dma_fence_enable_sw_signaling(q->lr.pfence);
> > >   		dma_fence_put(q->lr.pfence);
> > > -- 
> > > 2.43.0
> > > 
> 

Re: [RFC 03/14] drm/xe/pxp: Allocate PXP execution resources

[Daniele Ceraolo Spurio]

On 7/12/2024 3:43 PM, Matthew Brost wrote:
> On Fri, Jul 12, 2024 at 02:28:47PM -0700, Daniele Ceraolo Spurio wrote:
>> PXP requires submissions to the HW for the following operations
>>
>> 1) Key invalidation, done via the VCS engine
>> 2) Communication with the GSC FW for session management, done via the
>>     GSCCS
>>
>> For #1 we can allocate a simple kernel context, but #2 requires the
>> submissions to be done with PPGTT, which is not currently supported in Xe.
>> To add this support, the following changes have been included:
>>
>> - a new type of kernel-owned VM (marked as GSC)
>> - a new function to map a BO into a VM from within the kernel
>>
>> RFC note: I've tweaked some of the VM functions to return the fence
>> further up the stack, so I can wait on it from the PXP code. Not sure if
>> this is the best approach.
>>
>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>> Cc: Matthew Brost <matthew.brost@intel.com>
> Not a complete review but adding some thoughts. Looks sane enough to me.
>
> Random musing and nits below.
>
>> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> ---
>>   drivers/gpu/drm/xe/Makefile                   |   1 +
>>   drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |   7 +
>>   drivers/gpu/drm/xe/xe_exec_queue.c            |   3 +
>>   drivers/gpu/drm/xe/xe_pxp.c                   |  25 ++-
>>   drivers/gpu/drm/xe/xe_pxp_submit.c            | 188 ++++++++++++++++++
>>   drivers/gpu/drm/xe/xe_pxp_submit.h            |  16 ++
>>   drivers/gpu/drm/xe/xe_pxp_types.h             |  33 +++
>>   drivers/gpu/drm/xe/xe_vm.c                    | 100 +++++++++-
>>   drivers/gpu/drm/xe/xe_vm.h                    |   6 +
>>   drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
>>   10 files changed, 372 insertions(+), 8 deletions(-)
>>   create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
>>   create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h
>>
>> diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
>> index 5f15e6dd5057..a4514265085b 100644
>> --- a/drivers/gpu/drm/xe/Makefile
>> +++ b/drivers/gpu/drm/xe/Makefile
>> @@ -105,6 +105,7 @@ xe-y += xe_bb.o \
>>   	xe_pt.o \
>>   	xe_pt_walk.o \
>>   	xe_pxp.o \
>> +	xe_pxp_submit.o \
>>   	xe_query.o \
>>   	xe_range_fence.o \
>>   	xe_reg_sr.o \
>> diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
>> index 57520809e48d..f3c4cf10ba20 100644
>> --- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
>> +++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
>> @@ -6,6 +6,7 @@
>>   #ifndef _ABI_GSC_PXP_COMMANDS_ABI_H
>>   #define _ABI_GSC_PXP_COMMANDS_ABI_H
>>   
>> +#include <linux/sizes.h>
>>   #include <linux/types.h>
>>   
>>   /* Heci client ID for PXP commands */
>> @@ -13,6 +14,12 @@
>>   
>>   #define PXP_APIVER(x, y) (((x) & 0xFFFF) << 16 | ((y) & 0xFFFF))
>>   
>> +/*
>> + * A PXP sub-section in an HECI packet can be up to 64K big in each direction.
>> + * This does not include the top-level GSC header.
>> + */
>> +#define PXP_MAX_PACKET_SIZE SZ_64K
>> +
>>   /*
>>    * there are a lot of status codes for PXP, but we only define the cross-API
>>    * common ones that we actually can handle in the kernel driver. Other failure
>> diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
>> index 0ba37835849b..bc6e867aba17 100644
>> --- a/drivers/gpu/drm/xe/xe_exec_queue.c
>> +++ b/drivers/gpu/drm/xe/xe_exec_queue.c
>> @@ -131,6 +131,9 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
>>   	struct xe_exec_queue *q;
>>   	int err;
>>   
>> +	/* VMs for GSCCS queues (and only those) must have the XE_VM_FLAG_GSC flag */
>> +	xe_assert(xe, !vm || (!!(vm->flags & XE_VM_FLAG_GSC) == !!(hwe->engine_id == XE_HW_ENGINE_GSCCS0)));
>> +
> We should be able to remove this soon. More on that below.
>
>>   	q = __xe_exec_queue_alloc(xe, vm, logical_mask, width, hwe, flags,
>>   				  extensions);
>>   	if (IS_ERR(q))
>> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
>> index cdb29b104006..01386b9f0c50 100644
>> --- a/drivers/gpu/drm/xe/xe_pxp.c
>> +++ b/drivers/gpu/drm/xe/xe_pxp.c
>> @@ -12,6 +12,7 @@
>>   #include "xe_gt.h"
>>   #include "xe_gt_types.h"
>>   #include "xe_mmio.h"
>> +#include "xe_pxp_submit.h"
>>   #include "xe_pxp_types.h"
>>   #include "xe_uc_fw.h"
>>   #include "regs/xe_pxp_regs.h"
>> @@ -50,6 +51,20 @@ static int kcr_pxp_enable(const struct xe_pxp *pxp)
>>   	return kcr_pxp_set_status(pxp, true);
>>   }
>>   
>> +static int kcr_pxp_disable(const struct xe_pxp *pxp)
>> +{
>> +	return kcr_pxp_set_status(pxp, false);
>> +}
>> +
>> +static void pxp_fini(void *arg)
>> +{
>> +	struct xe_pxp *pxp = arg;
>> +
>> +	xe_pxp_destroy_execution_resources(pxp);
>> +
>> +	/* no need to explicitly disable KCR since we're going to do an FLR */
>> +}
>> +
>>   /**
>>    * xe_pxp_init - initialize PXP support
>>    * @xe: the xe_device structure
>> @@ -97,7 +112,15 @@ int xe_pxp_init(struct xe_device *xe)
>>   	if (err)
>>   		return err;
>>   
>> +	err = xe_pxp_allocate_execution_resources(pxp);
>> +	if (err)
>> +		goto kcr_disable;
>> +
>>   	xe->pxp = pxp;
>>   
>> -	return 0;
>> +	return devm_add_action_or_reset(xe->drm.dev, pxp_fini, pxp);
>> +
>> +kcr_disable:
>> +	kcr_pxp_disable(pxp);
>> +	return err;
>>   }
>> diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
>> new file mode 100644
>> index 000000000000..4fc3c7c58101
>> --- /dev/null
>> +++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
>> @@ -0,0 +1,188 @@
>> +// SPDX-License-Identifier: MIT
>> +/*
>> + * Copyright(c) 2024 Intel Corporation.
>> + */
>> +
>> +#include "xe_pxp_submit.h"
>> +
>> +#include <drm/xe_drm.h>
>> +
>> +#include "xe_device_types.h"
>> +#include "xe_bo.h"
>> +#include "xe_exec_queue.h"
>> +#include "xe_gsc_submit.h"
>> +#include "xe_gt.h"
>> +#include "xe_pxp_types.h"
>> +#include "xe_vm.h"
>> +#include "regs/xe_gt_regs.h"
>> +
>> +static int create_vcs_context(struct xe_pxp *pxp)
>> +{
>> +	struct xe_gt *gt = pxp->gt;
>> +	struct xe_hw_engine *hwe;
>> +	struct xe_exec_queue *q;
>> +
>> +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_VIDEO_DECODE, 0, true);
>> +	if (!hwe)
>> +		return -ENODEV;
>> +
> Ugh, really want to completely decouple an exec queue from hwe (e.g.
> don't pass in hwe to xe_exec_queue_create). I guess this already in code
> so fine here just a reminder of this ugliness.
>
>> +	q = xe_exec_queue_create(pxp->xe, NULL, BIT(hwe->logical_instance), 1, hwe,
>> +				 EXEC_QUEUE_FLAG_KERNEL | EXEC_QUEUE_FLAG_PERMANENT, 0);
>> +	if (IS_ERR(q))
>> +		return PTR_ERR(q);
>> +
>> +	pxp->vcs_queue = q;
>> +
> So how is this used? Not attached to a VM? GGTT or ring instructions
> only? Any downside of attaching this to GSC VM?

Ring instruction only, yes; we only use it to submit a key termination 
(next patch in the series).
I've made the GSC_VM only usable with the GSCCS so I didn't have to care 
about potentially having a kernel-owned non-faulting VM on 
user-accessible engines, where userspace might instead want to use a 
faulting VM. If we're removing the limitation and allowing the 2 types 
to mix that limitation for the GSC_VM should go away.

>
>> +	return 0;
>> +}
>> +
>> +static void destroy_vcs_context(struct xe_pxp *pxp)
>> +{
>> +	if (pxp->vcs_queue)
>> +		xe_exec_queue_put(pxp->vcs_queue);
>> +}
>> +
>> +/*
>> + * We allocate a single object for the batch and the input and output BOs. PXP
>> + * commands can require a lot of BO space (see PXP_MAX_PACKET_SIZE), but we
>> + * currently only support a subset of commands that are small (< 20 dwords),
>> + * so a single page is enough for now.
>> + */
>> +#define PXP_BB_SIZE		XE_PAGE_SIZE
>> +#define PXP_INOUT_SIZE		XE_PAGE_SIZE
>> +#define PXP_BO_SIZE		(PXP_BB_SIZE + (2 * PXP_INOUT_SIZE))
>> +#define PXP_BB_OFFSET		0
>> +#define PXP_MSG_IN_OFFSET 	PXP_BB_SIZE
>> +#define PXP_MSG_OUT_OFFSET 	(PXP_MSG_IN_OFFSET + PXP_INOUT_SIZE)
>> +static int allocate_gsc_execution_resources(struct xe_pxp *pxp)
>> +{
>> +	struct xe_gt *gt = pxp->gt;
>> +	struct xe_tile *tile = gt_to_tile(gt);
>> +	struct xe_device *xe = pxp->xe;
>> +	struct xe_hw_engine *hwe;
>> +	struct xe_vm *vm;
>> +	struct xe_bo *bo;
>> +	struct xe_exec_queue *q;
>> +	struct dma_fence *fence;
>> +	long timeout;
>> +	int err = 0;
>> +
>> +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_OTHER, OTHER_GSC_INSTANCE, false);
>> +
>> +	/* we shouldn't reach here if the GSC engine is not available */
>> +	xe_assert(xe, hwe);
>> +
>> +	/* PXP instructions must be issued from PPGTT */
>> +	vm = xe_vm_create(xe, XE_VM_FLAG_GSC);
>> +	if (IS_ERR(vm))
>> +		return PTR_ERR(vm);
>> +
>> +	/* We allocate a single object for the batch and the in/out memory */
>> +	xe_vm_lock(vm, false);
>> +	bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BO_SIZE, ttm_bo_type_kernel,
>> +				  XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_NEEDS_UC);
>> +	xe_vm_unlock(vm);
>> +	if (IS_ERR(bo)) {
>> +		err = PTR_ERR(bo);
>> +		goto vm_out;
>> +	}
>> +
>> +	fence = xe_vm_bind_bo(vm, bo, NULL, 0, XE_CACHE_WB);
>> +	if (IS_ERR(fence)) {
>> +		err = PTR_ERR(fence);
>> +		goto bo_out;
>> +	}
>> +
>> +	timeout = dma_fence_wait_timeout(fence, false, HZ);
>> +	dma_fence_put(fence);
>> +	if (timeout <= 0) {
>> +		err = timeout ?: -ETIME;
>> +		goto bo_out;
>> +	}
>> +
>> +	q = xe_exec_queue_create(xe, vm, BIT(hwe->logical_instance), 1, hwe,
>> +				 EXEC_QUEUE_FLAG_KERNEL |
>> +				 EXEC_QUEUE_FLAG_PERMANENT, 0);
>> +	if (IS_ERR(q)) {
>> +		err = PTR_ERR(q);
>> +		goto bo_out;
>> +	}
>> +
>> +	pxp->gsc_exec.vm = vm;
>> +	pxp->gsc_exec.bo = bo;
>> +	pxp->gsc_exec.batch = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_BB_OFFSET);
>> +	pxp->gsc_exec.msg_in = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_IN_OFFSET);
>> +	pxp->gsc_exec.msg_out = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_OUT_OFFSET);
> So with this mapping, all GSC are serially executed and waited on. There
> won't ever be a need to pipeline things? If the later is true you could
> xe_bb_* plus suballocation of the BO you map. More complex so if serial
> execute is all you will ever need, then yea probably don't use that.

We only send 2 types of commands, session initialization and session 
invalidation, which have to be serialized.

Even if we had other commands, the GSC is weird and submissions to it 
can complete with a "wait a bit then try again" message, so we have to 
wait until the fence is signaled, then check the memory and only if the 
memory has a "success" return we can move on to the next submission.

>
>> +	pxp->gsc_exec.q = q;
>> +
>> +	/* initialize host-session-handle (for all Xe-to-gsc-firmware PXP cmds) */
>> +	pxp->gsc_exec.host_session_handle = xe_gsc_create_host_session_id();
>> +
>> +	return 0;
>> +
>> +bo_out:
>> +	xe_vm_lock(vm, false);
>> +	xe_bo_unpin(bo);
>> +	xe_vm_unlock(vm);
>> +
>> +	xe_bo_put(bo);
> Can use helper I mention below.
>
>> +vm_out:
>> +	xe_vm_close_and_put(vm);
>> +
>> +	return err;
>> +}
>> +
>> +static void destroy_gsc_execution_resources(struct xe_pxp *pxp)
>> +{
>> +	if (!pxp->gsc_exec.q)
>> +		return;
>> +
>> +	iosys_map_clear(&pxp->gsc_exec.msg_out);
>> +	iosys_map_clear(&pxp->gsc_exec.msg_in);
>> +	iosys_map_clear(&pxp->gsc_exec.batch);
> I don't think this is strickly need as it just sets a pointer to NULL.
>
>> +
>> +	xe_exec_queue_put(pxp->gsc_exec.q);
>> +
>> +	xe_vm_lock(pxp->gsc_exec.vm, false);
>> +	xe_bo_unpin(pxp->gsc_exec.bo);
>> +	xe_vm_unlock(pxp->gsc_exec.vm);
>> +	xe_bo_put(pxp->gsc_exec.bo);
>> +
> This looks awfully like xe_bo_unpin_map_no_vm. Maybe rename that
> function and just use it?
>
> If a BO is private to a VM (this one is, xe_bo_lock and xe_vm_lock mean
> the same thing).

I didn't know the 2 locks where equivalent. I'll switch to the helper.

>
>> +	xe_vm_close_and_put(pxp->gsc_exec.vm);
>> +}
>> +
>> +/**
>> + * xe_pxp_allocate_execution_resources - Allocate PXP submission objects
>> + * @pxp: the xe_pxp structure
>> + *
>> + * Allocates exec_queues objects for VCS and GSCCS submission. The GSCCS
>> + * submissions are done via PPGTT, so this function allocates a VM for it and
>> + * maps the object into it.
>> + *
>> + * Returns 0 if the allocation and mapping is successful, an errno value
>> + * otherwise.
>> + */
>> +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp)
>> +{
>> +	int err;
>> +
>> +	err = create_vcs_context(pxp);
>> +	if (err)
>> +		return err;
>> +
>> +	err = allocate_gsc_execution_resources(pxp);
>> +	if (err)
>> +		goto destroy_vcs_context;
>> +
>> +	return 0;
>> +
>> +destroy_vcs_context:
>> +	destroy_vcs_context(pxp);
>> +	return err;
>> +}
>> +
>> +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp)
>> +{
>> +	destroy_gsc_execution_resources(pxp);
>> +	destroy_vcs_context(pxp);
>> +}
>> diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
>> new file mode 100644
>> index 000000000000..1a971fadc081
>> --- /dev/null
>> +++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
>> @@ -0,0 +1,16 @@
>> +/* SPDX-License-Identifier: MIT */
>> +/*
>> + * Copyright(c) 2024, Intel Corporation. All rights reserved.
>> + */
>> +
>> +#ifndef __XE_PXP_SUBMIT_H__
>> +#define __XE_PXP_SUBMIT_H__
>> +
>> +#include <linux/types.h>
>> +
>> +struct xe_pxp;
>> +
>> +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
>> +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
>> +
>> +#endif /* __XE_PXP_SUBMIT_H__ */
>> diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
>> index 1561e3bd2676..c16813253b47 100644
>> --- a/drivers/gpu/drm/xe/xe_pxp_types.h
>> +++ b/drivers/gpu/drm/xe/xe_pxp_types.h
>> @@ -6,10 +6,14 @@
>>   #ifndef __XE_PXP_TYPES_H__
>>   #define __XE_PXP_TYPES_H__
>>   
>> +#include <linux/iosys-map.h>
>>   #include <linux/types.h>
>>   
>> +struct xe_bo;
>> +struct xe_exec_queue;
>>   struct xe_device;
>>   struct xe_gt;
>> +struct xe_vm;
>>   
>>   /**
>>    * struct xe_pxp - pxp state
>> @@ -23,6 +27,35 @@ struct xe_pxp {
>>   	 * (VDBOX, KCR and GSC)
>>   	 */
>>   	struct xe_gt *gt;
>> +
>> +	/** @vcs_queue: kernel-owned VCS exec queue used for PXP operations */
>> +	struct xe_exec_queue *vcs_queue;
>> +
>> +	/** @gsc_exec: kernel-owned objects for PXP submissions to the GSCCS */
>> +	struct {
>> +		/**
>> +		 * @gsc_exec.host_session_handle: handle used in communications
>> +		 * with the GSC firmware.
>> +		 */
>> +		u64 host_session_handle;
>> +		/** @gsc_exec.vm: VM used for PXP submissions to the GSCCS */
>> +		struct xe_vm *vm;
>> +		/** @gsc_exec.q: GSCCS exec queue for PXP submissions */
>> +		struct xe_exec_queue *q;
>> +
>> +		/**
>> +		 * @gsc_exec.bo: BO used for submissions to the GSCCS and GSC
>> +		 * FW. It includes space for the GSCCS batch and the
>> +		 * input/output buffers read/written by the FW
>> +		 */
>> +		struct xe_bo *bo;
>> +		/** @gsc_exec.batch: iosys_map to the batch memory within the BO */
>> +		struct iosys_map batch;
>> +		/** @gsc_exec.msg_in: iosys_map to the input memory within the BO */
>> +		struct iosys_map msg_in;
>> +		/** @gsc_exec.msg_out: iosys_map to the output memory within the BO */
>> +		struct iosys_map msg_out;
>> +	} gsc_exec;
>>   };
>>   
>>   #endif /* __XE_PXP_TYPES_H__ */
>> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
>> index 02f684c0330d..412ec9cb9650 100644
>> --- a/drivers/gpu/drm/xe/xe_vm.c
>> +++ b/drivers/gpu/drm/xe/xe_vm.c
>> @@ -1315,6 +1315,15 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
>>   	struct xe_tile *tile;
>>   	u8 id;
>>   
>> +	/*
>> +	 * All GSC VMs are owned by the kernel and can also only be used on
>> +	 * the GSCCS. We don't want a kernel-owned VM to put the device in
>> +	 * either fault or not fault mode, so we need to exclude the GSC VMs
>> +	 * from that count; this is only safe if we ensure that all GSC VMs are
>> +	 * non-faulting.
>> +	 */
>> +	xe_assert(xe, !((flags & XE_VM_FLAG_GSC) && (flags & XE_VM_FLAG_FAULT_MODE)));
>> +
>>   	vm = kzalloc(sizeof(*vm), GFP_KERNEL);
>>   	if (!vm)
>>   		return ERR_PTR(-ENOMEM);
>> @@ -1442,7 +1451,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
>>   	mutex_lock(&xe->usm.lock);
>>   	if (flags & XE_VM_FLAG_FAULT_MODE)
>>   		xe->usm.num_vm_in_fault_mode++;
>> -	else if (!(flags & XE_VM_FLAG_MIGRATION))
>> +	else if (!(flags & (XE_VM_FLAG_MIGRATION | XE_VM_FLAG_GSC)))
> This change is good now but should become unnecessary once Francois
> lands some code to remove the restriction of mixing faulting and
> non-faulting VM within a device.
>
>>   		xe->usm.num_vm_in_non_fault_mode++;
>>   	mutex_unlock(&xe->usm.lock);
>>   
>> @@ -2867,11 +2876,10 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
>>   	for (i = 0; i < vops->num_syncs; i++)
>>   		xe_sync_entry_signal(vops->syncs + i, fence);
>>   	xe_exec_queue_last_fence_set(wait_exec_queue, vm, fence);
>> -	dma_fence_put(fence);
> Nit: I'd send this change and associated change in xe_vm_bind_ioctl +
> vm_bind_ioctl_ops_execute in its own patch, perhaps even as an
> independent series which I'd RB immediately.
>
> Change looks good though and could be useful else where too.
>
>>   }
>>   
>> -static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>> -				     struct xe_vma_ops *vops)
>> +static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>> +						   struct xe_vma_ops *vops)
>>   {
>>   	struct drm_exec exec;
>>   	struct dma_fence *fence;
>> @@ -2889,7 +2897,6 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>>   
>>   		fence = ops_execute(vm, vops);
>>   		if (IS_ERR(fence)) {
>> -			err = PTR_ERR(fence);
>>   			/* FIXME: Killing VM rather than proper error handling */
>>   			xe_vm_kill(vm, false);
> Looks like you are on old baseline before this series landed [1]. I
> suggest rebasing as those changes creep up in the upper layers a bit.
>
> [1] https://patchwork.freedesktop.org/series/133034/

Yes, my local tree is from last week. I'll rebase and split out the 
changes to their own patch as suggested.

>>   			goto unlock;
>> @@ -2900,7 +2907,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
>>   
>>   unlock:
>>   	drm_exec_fini(&exec);
>> -	return err;
>> +	return fence;
>>   }
>>   
>>   #define SUPPORTED_FLAGS	\
>> @@ -3114,6 +3121,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>   	struct xe_sync_entry *syncs = NULL;
>>   	struct drm_xe_vm_bind_op *bind_ops;
>>   	struct xe_vma_ops vops;
>> +	struct dma_fence *fence;
>>   	int err;
>>   	int i;
>>   
>> @@ -3264,7 +3272,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>   		goto unwind_ops;
>>   	}
>>   
>> -	err = vm_bind_ioctl_ops_execute(vm, &vops);
>> +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
>> +	if (IS_ERR(fence))
>> +		err = PTR_ERR(fence);
>> +	else
>> +		dma_fence_put(fence);
>>   
>>   unwind_ops:
>>   	if (err && err != -ENODATA)
>> @@ -3297,6 +3309,80 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>   	return err;
>>   }
>>   
>> +/**
>> + * xe_vm_bind_bo - bind a kernel BO to a VM
>> + * @vm: VM to bind the BO to
>> + * @bo: BO to bind
>> + * @q: exec queue to use for the bind (optional)
>> + * @addr: address at which to bind the BO
>> + * @cache_lvl: PAT cache level to use
>> + *
>> + * Execute a VM bind map operation on a kernel-owned BO to bind it into a
>> + * kernel-owned VM.
>> + *
>> + * Returns 0 if the ops execution is successful, an errno value otherwise.
>> + * TODO: return a fence instead.
>> + */
>> +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
>> +				struct xe_exec_queue *q, u64 addr,
>> +				enum xe_cache_level cache_lvl)
>> +{
>> +	struct xe_vma_ops vops;
>> +	struct drm_gpuva_ops *ops = NULL;
>> +	struct dma_fence *fence;
>> +	int err;
>> +
>> +	xe_bo_get(bo);
>> +	xe_vm_get(vm);
>> +	if (q)
>> +		xe_exec_queue_get(q);
>> +
>> +	down_write(&vm->lock);
>> +
>> +	xe_vma_ops_init(&vops, vm, q, NULL, 0);
>> +
>> +	ops = vm_bind_ioctl_ops_create(vm, bo, 0, addr, bo->size,
>> +				       DRM_XE_VM_BIND_OP_MAP, 0,
>> +				       vm->xe->pat.idx[cache_lvl], 0);
>> +	if (IS_ERR(ops)) {
>> +		err = PTR_ERR(ops);
>> +		goto release_vm_lock;
>> +	}
>> +
>> +	err = vm_bind_ioctl_ops_parse(vm, q, ops, NULL, 0, &vops, true);
>> +	if (err)
>> +		goto release_vm_lock;
>> +
>> +	/* Nothing to do */
>> +	if (list_empty(&vops.list)) {
> Can this ever be true? In the current usage it appear so. Maybe convert
> to an asset !list_empty to simplify this function slightly?

will do.

Daniele

>
> Matt
>
>> +		err = -ENODATA;
>> +		goto unwind_ops;
>> +	}
>> +
>> +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
>> +	if (IS_ERR(fence))
>> +		err = PTR_ERR(fence);
>> +
>> +unwind_ops:
>> +	if (err && err != -ENODATA)
>> +		vm_bind_ioctl_ops_unwind(vm, &ops, 1);
>> +
>> +	drm_gpuva_ops_free(&vm->gpuvm, ops);
>> +
>> +release_vm_lock:
>> +	up_write(&vm->lock);
>> +
>> +	if (q)
>> +		xe_exec_queue_put(q);
>> +	xe_vm_put(vm);
>> +	xe_bo_put(bo);
>> +
>> +	if (err)
>> +		fence = ERR_PTR(err);
>> +
>> +	return fence;
>> +}
>> +
>>   /**
>>    * xe_vm_lock() - Lock the vm's dma_resv object
>>    * @vm: The struct xe_vm whose lock is to be locked
>> diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
>> index b481608b12f1..5e298ac90dfc 100644
>> --- a/drivers/gpu/drm/xe/xe_vm.h
>> +++ b/drivers/gpu/drm/xe/xe_vm.h
>> @@ -19,6 +19,8 @@ struct drm_file;
>>   struct ttm_buffer_object;
>>   struct ttm_validate_buffer;
>>   
>> +struct dma_fence;
>> +
>>   struct xe_exec_queue;
>>   struct xe_file;
>>   struct xe_sync_entry;
>> @@ -248,6 +250,10 @@ int xe_vm_lock_vma(struct drm_exec *exec, struct xe_vma *vma);
>>   int xe_vm_validate_rebind(struct xe_vm *vm, struct drm_exec *exec,
>>   			  unsigned int num_fences);
>>   
>> +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
>> +				struct xe_exec_queue *q, u64 addr,
>> +				enum xe_cache_level cache_lvl);
>> +
>>   /**
>>    * xe_vm_resv() - Return's the vm's reservation object
>>    * @vm: The vm
>> diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
>> index ce1a63a5e3e7..60ce327d303c 100644
>> --- a/drivers/gpu/drm/xe/xe_vm_types.h
>> +++ b/drivers/gpu/drm/xe/xe_vm_types.h
>> @@ -152,6 +152,7 @@ struct xe_vm {
>>   #define XE_VM_FLAG_BANNED		BIT(5)
>>   #define XE_VM_FLAG_TILE_ID(flags)	FIELD_GET(GENMASK(7, 6), flags)
>>   #define XE_VM_FLAG_SET_TILE_ID(tile)	FIELD_PREP(GENMASK(7, 6), (tile)->id)
>> +#define XE_VM_FLAG_GSC			BIT(8)
>>   	unsigned long flags;
>>   
>>   	/** @composite_fence_ctx: context composite fence */
>> -- 
>> 2.43.0
>>

Re: [RFC 03/14] drm/xe/pxp: Allocate PXP execution resources

[Matthew Brost]

On Fri, Jul 12, 2024 at 04:00:09PM -0700, Daniele Ceraolo Spurio wrote:
> 
> 
> On 7/12/2024 3:43 PM, Matthew Brost wrote:
> > On Fri, Jul 12, 2024 at 02:28:47PM -0700, Daniele Ceraolo Spurio wrote:
> > > PXP requires submissions to the HW for the following operations
> > > 
> > > 1) Key invalidation, done via the VCS engine
> > > 2) Communication with the GSC FW for session management, done via the
> > >     GSCCS
> > > 
> > > For #1 we can allocate a simple kernel context, but #2 requires the
> > > submissions to be done with PPGTT, which is not currently supported in Xe.
> > > To add this support, the following changes have been included:
> > > 
> > > - a new type of kernel-owned VM (marked as GSC)
> > > - a new function to map a BO into a VM from within the kernel
> > > 
> > > RFC note: I've tweaked some of the VM functions to return the fence
> > > further up the stack, so I can wait on it from the PXP code. Not sure if
> > > this is the best approach.
> > > 
> > > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> > > Cc: Matthew Brost <matthew.brost@intel.com>
> > Not a complete review but adding some thoughts. Looks sane enough to me.
> > 
> > Random musing and nits below.
> > 
> > > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > > ---
> > >   drivers/gpu/drm/xe/Makefile                   |   1 +
> > >   drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |   7 +
> > >   drivers/gpu/drm/xe/xe_exec_queue.c            |   3 +
> > >   drivers/gpu/drm/xe/xe_pxp.c                   |  25 ++-
> > >   drivers/gpu/drm/xe/xe_pxp_submit.c            | 188 ++++++++++++++++++
> > >   drivers/gpu/drm/xe/xe_pxp_submit.h            |  16 ++
> > >   drivers/gpu/drm/xe/xe_pxp_types.h             |  33 +++
> > >   drivers/gpu/drm/xe/xe_vm.c                    | 100 +++++++++-
> > >   drivers/gpu/drm/xe/xe_vm.h                    |   6 +
> > >   drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
> > >   10 files changed, 372 insertions(+), 8 deletions(-)
> > >   create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
> > >   create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h
> > > 
> > > diff --git a/drivers/gpu/drm/xe/Makefile b/drivers/gpu/drm/xe/Makefile
> > > index 5f15e6dd5057..a4514265085b 100644
> > > --- a/drivers/gpu/drm/xe/Makefile
> > > +++ b/drivers/gpu/drm/xe/Makefile
> > > @@ -105,6 +105,7 @@ xe-y += xe_bb.o \
> > >   	xe_pt.o \
> > >   	xe_pt_walk.o \
> > >   	xe_pxp.o \
> > > +	xe_pxp_submit.o \
> > >   	xe_query.o \
> > >   	xe_range_fence.o \
> > >   	xe_reg_sr.o \
> > > diff --git a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> > > index 57520809e48d..f3c4cf10ba20 100644
> > > --- a/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> > > +++ b/drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h
> > > @@ -6,6 +6,7 @@
> > >   #ifndef _ABI_GSC_PXP_COMMANDS_ABI_H
> > >   #define _ABI_GSC_PXP_COMMANDS_ABI_H
> > > +#include <linux/sizes.h>
> > >   #include <linux/types.h>
> > >   /* Heci client ID for PXP commands */
> > > @@ -13,6 +14,12 @@
> > >   #define PXP_APIVER(x, y) (((x) & 0xFFFF) << 16 | ((y) & 0xFFFF))
> > > +/*
> > > + * A PXP sub-section in an HECI packet can be up to 64K big in each direction.
> > > + * This does not include the top-level GSC header.
> > > + */
> > > +#define PXP_MAX_PACKET_SIZE SZ_64K
> > > +
> > >   /*
> > >    * there are a lot of status codes for PXP, but we only define the cross-API
> > >    * common ones that we actually can handle in the kernel driver. Other failure
> > > diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
> > > index 0ba37835849b..bc6e867aba17 100644
> > > --- a/drivers/gpu/drm/xe/xe_exec_queue.c
> > > +++ b/drivers/gpu/drm/xe/xe_exec_queue.c
> > > @@ -131,6 +131,9 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
> > >   	struct xe_exec_queue *q;
> > >   	int err;
> > > +	/* VMs for GSCCS queues (and only those) must have the XE_VM_FLAG_GSC flag */
> > > +	xe_assert(xe, !vm || (!!(vm->flags & XE_VM_FLAG_GSC) == !!(hwe->engine_id == XE_HW_ENGINE_GSCCS0)));
> > > +
> > We should be able to remove this soon. More on that below.
> > 
> > >   	q = __xe_exec_queue_alloc(xe, vm, logical_mask, width, hwe, flags,
> > >   				  extensions);
> > >   	if (IS_ERR(q))
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> > > index cdb29b104006..01386b9f0c50 100644
> > > --- a/drivers/gpu/drm/xe/xe_pxp.c
> > > +++ b/drivers/gpu/drm/xe/xe_pxp.c
> > > @@ -12,6 +12,7 @@
> > >   #include "xe_gt.h"
> > >   #include "xe_gt_types.h"
> > >   #include "xe_mmio.h"
> > > +#include "xe_pxp_submit.h"
> > >   #include "xe_pxp_types.h"
> > >   #include "xe_uc_fw.h"
> > >   #include "regs/xe_pxp_regs.h"
> > > @@ -50,6 +51,20 @@ static int kcr_pxp_enable(const struct xe_pxp *pxp)
> > >   	return kcr_pxp_set_status(pxp, true);
> > >   }
> > > +static int kcr_pxp_disable(const struct xe_pxp *pxp)
> > > +{
> > > +	return kcr_pxp_set_status(pxp, false);
> > > +}
> > > +
> > > +static void pxp_fini(void *arg)
> > > +{
> > > +	struct xe_pxp *pxp = arg;
> > > +
> > > +	xe_pxp_destroy_execution_resources(pxp);
> > > +
> > > +	/* no need to explicitly disable KCR since we're going to do an FLR */
> > > +}
> > > +
> > >   /**
> > >    * xe_pxp_init - initialize PXP support
> > >    * @xe: the xe_device structure
> > > @@ -97,7 +112,15 @@ int xe_pxp_init(struct xe_device *xe)
> > >   	if (err)
> > >   		return err;
> > > +	err = xe_pxp_allocate_execution_resources(pxp);
> > > +	if (err)
> > > +		goto kcr_disable;
> > > +
> > >   	xe->pxp = pxp;
> > > -	return 0;
> > > +	return devm_add_action_or_reset(xe->drm.dev, pxp_fini, pxp);
> > > +
> > > +kcr_disable:
> > > +	kcr_pxp_disable(pxp);
> > > +	return err;
> > >   }
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.c b/drivers/gpu/drm/xe/xe_pxp_submit.c
> > > new file mode 100644
> > > index 000000000000..4fc3c7c58101
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/xe/xe_pxp_submit.c
> > > @@ -0,0 +1,188 @@
> > > +// SPDX-License-Identifier: MIT
> > > +/*
> > > + * Copyright(c) 2024 Intel Corporation.
> > > + */
> > > +
> > > +#include "xe_pxp_submit.h"
> > > +
> > > +#include <drm/xe_drm.h>
> > > +
> > > +#include "xe_device_types.h"
> > > +#include "xe_bo.h"
> > > +#include "xe_exec_queue.h"
> > > +#include "xe_gsc_submit.h"
> > > +#include "xe_gt.h"
> > > +#include "xe_pxp_types.h"
> > > +#include "xe_vm.h"
> > > +#include "regs/xe_gt_regs.h"
> > > +
> > > +static int create_vcs_context(struct xe_pxp *pxp)
> > > +{
> > > +	struct xe_gt *gt = pxp->gt;
> > > +	struct xe_hw_engine *hwe;
> > > +	struct xe_exec_queue *q;
> > > +
> > > +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_VIDEO_DECODE, 0, true);
> > > +	if (!hwe)
> > > +		return -ENODEV;
> > > +
> > Ugh, really want to completely decouple an exec queue from hwe (e.g.
> > don't pass in hwe to xe_exec_queue_create). I guess this already in code
> > so fine here just a reminder of this ugliness.
> > 
> > > +	q = xe_exec_queue_create(pxp->xe, NULL, BIT(hwe->logical_instance), 1, hwe,
> > > +				 EXEC_QUEUE_FLAG_KERNEL | EXEC_QUEUE_FLAG_PERMANENT, 0);
> > > +	if (IS_ERR(q))
> > > +		return PTR_ERR(q);
> > > +
> > > +	pxp->vcs_queue = q;
> > > +
> > So how is this used? Not attached to a VM? GGTT or ring instructions
> > only? Any downside of attaching this to GSC VM?
> 
> Ring instruction only, yes; we only use it to submit a key termination (next
> patch in the series).
> I've made the GSC_VM only usable with the GSCCS so I didn't have to care
> about potentially having a kernel-owned non-faulting VM on user-accessible
> engines, where userspace might instead want to use a faulting VM. If we're
> removing the limitation and allowing the 2 types to mix that limitation for
> the GSC_VM should go away.

If this is going to be ring instructions only yea I guess leave it
without a VM as there will be a pentally to switch between fault mode
and non-fault mode. I think if we never export a dma-fence from the
vcs_queue queue we can bypass a mode switch. If you export a dma-fence
from the vcs_queue anywhere (install in sync obj, dma-resv, etc...) we
cannot without lockdep getting upset or potentially deadlocking
ourselves. We will need to double check with Thomas / Francois though.

Matt

> 
> > 
> > > +	return 0;
> > > +}
> > > +
> > > +static void destroy_vcs_context(struct xe_pxp *pxp)
> > > +{
> > > +	if (pxp->vcs_queue)
> > > +		xe_exec_queue_put(pxp->vcs_queue);
> > > +}
> > > +
> > > +/*
> > > + * We allocate a single object for the batch and the input and output BOs. PXP
> > > + * commands can require a lot of BO space (see PXP_MAX_PACKET_SIZE), but we
> > > + * currently only support a subset of commands that are small (< 20 dwords),
> > > + * so a single page is enough for now.
> > > + */
> > > +#define PXP_BB_SIZE		XE_PAGE_SIZE
> > > +#define PXP_INOUT_SIZE		XE_PAGE_SIZE
> > > +#define PXP_BO_SIZE		(PXP_BB_SIZE + (2 * PXP_INOUT_SIZE))
> > > +#define PXP_BB_OFFSET		0
> > > +#define PXP_MSG_IN_OFFSET 	PXP_BB_SIZE
> > > +#define PXP_MSG_OUT_OFFSET 	(PXP_MSG_IN_OFFSET + PXP_INOUT_SIZE)
> > > +static int allocate_gsc_execution_resources(struct xe_pxp *pxp)
> > > +{
> > > +	struct xe_gt *gt = pxp->gt;
> > > +	struct xe_tile *tile = gt_to_tile(gt);
> > > +	struct xe_device *xe = pxp->xe;
> > > +	struct xe_hw_engine *hwe;
> > > +	struct xe_vm *vm;
> > > +	struct xe_bo *bo;
> > > +	struct xe_exec_queue *q;
> > > +	struct dma_fence *fence;
> > > +	long timeout;
> > > +	int err = 0;
> > > +
> > > +	hwe = xe_gt_hw_engine(gt, XE_ENGINE_CLASS_OTHER, OTHER_GSC_INSTANCE, false);
> > > +
> > > +	/* we shouldn't reach here if the GSC engine is not available */
> > > +	xe_assert(xe, hwe);
> > > +
> > > +	/* PXP instructions must be issued from PPGTT */
> > > +	vm = xe_vm_create(xe, XE_VM_FLAG_GSC);
> > > +	if (IS_ERR(vm))
> > > +		return PTR_ERR(vm);
> > > +
> > > +	/* We allocate a single object for the batch and the in/out memory */
> > > +	xe_vm_lock(vm, false);
> > > +	bo = xe_bo_create_pin_map(xe, tile, vm, PXP_BO_SIZE, ttm_bo_type_kernel,
> > > +				  XE_BO_FLAG_SYSTEM | XE_BO_FLAG_PINNED | XE_BO_FLAG_NEEDS_UC);
> > > +	xe_vm_unlock(vm);
> > > +	if (IS_ERR(bo)) {
> > > +		err = PTR_ERR(bo);
> > > +		goto vm_out;
> > > +	}
> > > +
> > > +	fence = xe_vm_bind_bo(vm, bo, NULL, 0, XE_CACHE_WB);
> > > +	if (IS_ERR(fence)) {
> > > +		err = PTR_ERR(fence);
> > > +		goto bo_out;
> > > +	}
> > > +
> > > +	timeout = dma_fence_wait_timeout(fence, false, HZ);
> > > +	dma_fence_put(fence);
> > > +	if (timeout <= 0) {
> > > +		err = timeout ?: -ETIME;
> > > +		goto bo_out;
> > > +	}
> > > +
> > > +	q = xe_exec_queue_create(xe, vm, BIT(hwe->logical_instance), 1, hwe,
> > > +				 EXEC_QUEUE_FLAG_KERNEL |
> > > +				 EXEC_QUEUE_FLAG_PERMANENT, 0);
> > > +	if (IS_ERR(q)) {
> > > +		err = PTR_ERR(q);
> > > +		goto bo_out;
> > > +	}
> > > +
> > > +	pxp->gsc_exec.vm = vm;
> > > +	pxp->gsc_exec.bo = bo;
> > > +	pxp->gsc_exec.batch = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_BB_OFFSET);
> > > +	pxp->gsc_exec.msg_in = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_IN_OFFSET);
> > > +	pxp->gsc_exec.msg_out = IOSYS_MAP_INIT_OFFSET(&bo->vmap, PXP_MSG_OUT_OFFSET);
> > So with this mapping, all GSC are serially executed and waited on. There
> > won't ever be a need to pipeline things? If the later is true you could
> > xe_bb_* plus suballocation of the BO you map. More complex so if serial
> > execute is all you will ever need, then yea probably don't use that.
> 
> We only send 2 types of commands, session initialization and session
> invalidation, which have to be serialized.
> 
> Even if we had other commands, the GSC is weird and submissions to it can
> complete with a "wait a bit then try again" message, so we have to wait
> until the fence is signaled, then check the memory and only if the memory
> has a "success" return we can move on to the next submission.
> 
> > 
> > > +	pxp->gsc_exec.q = q;
> > > +
> > > +	/* initialize host-session-handle (for all Xe-to-gsc-firmware PXP cmds) */
> > > +	pxp->gsc_exec.host_session_handle = xe_gsc_create_host_session_id();
> > > +
> > > +	return 0;
> > > +
> > > +bo_out:
> > > +	xe_vm_lock(vm, false);
> > > +	xe_bo_unpin(bo);
> > > +	xe_vm_unlock(vm);
> > > +
> > > +	xe_bo_put(bo);
> > Can use helper I mention below.
> > 
> > > +vm_out:
> > > +	xe_vm_close_and_put(vm);
> > > +
> > > +	return err;
> > > +}
> > > +
> > > +static void destroy_gsc_execution_resources(struct xe_pxp *pxp)
> > > +{
> > > +	if (!pxp->gsc_exec.q)
> > > +		return;
> > > +
> > > +	iosys_map_clear(&pxp->gsc_exec.msg_out);
> > > +	iosys_map_clear(&pxp->gsc_exec.msg_in);
> > > +	iosys_map_clear(&pxp->gsc_exec.batch);
> > I don't think this is strickly need as it just sets a pointer to NULL.
> > 
> > > +
> > > +	xe_exec_queue_put(pxp->gsc_exec.q);
> > > +
> > > +	xe_vm_lock(pxp->gsc_exec.vm, false);
> > > +	xe_bo_unpin(pxp->gsc_exec.bo);
> > > +	xe_vm_unlock(pxp->gsc_exec.vm);
> > > +	xe_bo_put(pxp->gsc_exec.bo);
> > > +
> > This looks awfully like xe_bo_unpin_map_no_vm. Maybe rename that
> > function and just use it?
> > 
> > If a BO is private to a VM (this one is, xe_bo_lock and xe_vm_lock mean
> > the same thing).
> 
> I didn't know the 2 locks where equivalent. I'll switch to the helper.
> 
> > 
> > > +	xe_vm_close_and_put(pxp->gsc_exec.vm);
> > > +}
> > > +
> > > +/**
> > > + * xe_pxp_allocate_execution_resources - Allocate PXP submission objects
> > > + * @pxp: the xe_pxp structure
> > > + *
> > > + * Allocates exec_queues objects for VCS and GSCCS submission. The GSCCS
> > > + * submissions are done via PPGTT, so this function allocates a VM for it and
> > > + * maps the object into it.
> > > + *
> > > + * Returns 0 if the allocation and mapping is successful, an errno value
> > > + * otherwise.
> > > + */
> > > +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp)
> > > +{
> > > +	int err;
> > > +
> > > +	err = create_vcs_context(pxp);
> > > +	if (err)
> > > +		return err;
> > > +
> > > +	err = allocate_gsc_execution_resources(pxp);
> > > +	if (err)
> > > +		goto destroy_vcs_context;
> > > +
> > > +	return 0;
> > > +
> > > +destroy_vcs_context:
> > > +	destroy_vcs_context(pxp);
> > > +	return err;
> > > +}
> > > +
> > > +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp)
> > > +{
> > > +	destroy_gsc_execution_resources(pxp);
> > > +	destroy_vcs_context(pxp);
> > > +}
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp_submit.h b/drivers/gpu/drm/xe/xe_pxp_submit.h
> > > new file mode 100644
> > > index 000000000000..1a971fadc081
> > > --- /dev/null
> > > +++ b/drivers/gpu/drm/xe/xe_pxp_submit.h
> > > @@ -0,0 +1,16 @@
> > > +/* SPDX-License-Identifier: MIT */
> > > +/*
> > > + * Copyright(c) 2024, Intel Corporation. All rights reserved.
> > > + */
> > > +
> > > +#ifndef __XE_PXP_SUBMIT_H__
> > > +#define __XE_PXP_SUBMIT_H__
> > > +
> > > +#include <linux/types.h>
> > > +
> > > +struct xe_pxp;
> > > +
> > > +int xe_pxp_allocate_execution_resources(struct xe_pxp *pxp);
> > > +void xe_pxp_destroy_execution_resources(struct xe_pxp *pxp);
> > > +
> > > +#endif /* __XE_PXP_SUBMIT_H__ */
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp_types.h b/drivers/gpu/drm/xe/xe_pxp_types.h
> > > index 1561e3bd2676..c16813253b47 100644
> > > --- a/drivers/gpu/drm/xe/xe_pxp_types.h
> > > +++ b/drivers/gpu/drm/xe/xe_pxp_types.h
> > > @@ -6,10 +6,14 @@
> > >   #ifndef __XE_PXP_TYPES_H__
> > >   #define __XE_PXP_TYPES_H__
> > > +#include <linux/iosys-map.h>
> > >   #include <linux/types.h>
> > > +struct xe_bo;
> > > +struct xe_exec_queue;
> > >   struct xe_device;
> > >   struct xe_gt;
> > > +struct xe_vm;
> > >   /**
> > >    * struct xe_pxp - pxp state
> > > @@ -23,6 +27,35 @@ struct xe_pxp {
> > >   	 * (VDBOX, KCR and GSC)
> > >   	 */
> > >   	struct xe_gt *gt;
> > > +
> > > +	/** @vcs_queue: kernel-owned VCS exec queue used for PXP operations */
> > > +	struct xe_exec_queue *vcs_queue;
> > > +
> > > +	/** @gsc_exec: kernel-owned objects for PXP submissions to the GSCCS */
> > > +	struct {
> > > +		/**
> > > +		 * @gsc_exec.host_session_handle: handle used in communications
> > > +		 * with the GSC firmware.
> > > +		 */
> > > +		u64 host_session_handle;
> > > +		/** @gsc_exec.vm: VM used for PXP submissions to the GSCCS */
> > > +		struct xe_vm *vm;
> > > +		/** @gsc_exec.q: GSCCS exec queue for PXP submissions */
> > > +		struct xe_exec_queue *q;
> > > +
> > > +		/**
> > > +		 * @gsc_exec.bo: BO used for submissions to the GSCCS and GSC
> > > +		 * FW. It includes space for the GSCCS batch and the
> > > +		 * input/output buffers read/written by the FW
> > > +		 */
> > > +		struct xe_bo *bo;
> > > +		/** @gsc_exec.batch: iosys_map to the batch memory within the BO */
> > > +		struct iosys_map batch;
> > > +		/** @gsc_exec.msg_in: iosys_map to the input memory within the BO */
> > > +		struct iosys_map msg_in;
> > > +		/** @gsc_exec.msg_out: iosys_map to the output memory within the BO */
> > > +		struct iosys_map msg_out;
> > > +	} gsc_exec;
> > >   };
> > >   #endif /* __XE_PXP_TYPES_H__ */
> > > diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> > > index 02f684c0330d..412ec9cb9650 100644
> > > --- a/drivers/gpu/drm/xe/xe_vm.c
> > > +++ b/drivers/gpu/drm/xe/xe_vm.c
> > > @@ -1315,6 +1315,15 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
> > >   	struct xe_tile *tile;
> > >   	u8 id;
> > > +	/*
> > > +	 * All GSC VMs are owned by the kernel and can also only be used on
> > > +	 * the GSCCS. We don't want a kernel-owned VM to put the device in
> > > +	 * either fault or not fault mode, so we need to exclude the GSC VMs
> > > +	 * from that count; this is only safe if we ensure that all GSC VMs are
> > > +	 * non-faulting.
> > > +	 */
> > > +	xe_assert(xe, !((flags & XE_VM_FLAG_GSC) && (flags & XE_VM_FLAG_FAULT_MODE)));
> > > +
> > >   	vm = kzalloc(sizeof(*vm), GFP_KERNEL);
> > >   	if (!vm)
> > >   		return ERR_PTR(-ENOMEM);
> > > @@ -1442,7 +1451,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)
> > >   	mutex_lock(&xe->usm.lock);
> > >   	if (flags & XE_VM_FLAG_FAULT_MODE)
> > >   		xe->usm.num_vm_in_fault_mode++;
> > > -	else if (!(flags & XE_VM_FLAG_MIGRATION))
> > > +	else if (!(flags & (XE_VM_FLAG_MIGRATION | XE_VM_FLAG_GSC)))
> > This change is good now but should become unnecessary once Francois
> > lands some code to remove the restriction of mixing faulting and
> > non-faulting VM within a device.
> > 
> > >   		xe->usm.num_vm_in_non_fault_mode++;
> > >   	mutex_unlock(&xe->usm.lock);
> > > @@ -2867,11 +2876,10 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
> > >   	for (i = 0; i < vops->num_syncs; i++)
> > >   		xe_sync_entry_signal(vops->syncs + i, fence);
> > >   	xe_exec_queue_last_fence_set(wait_exec_queue, vm, fence);
> > > -	dma_fence_put(fence);
> > Nit: I'd send this change and associated change in xe_vm_bind_ioctl +
> > vm_bind_ioctl_ops_execute in its own patch, perhaps even as an
> > independent series which I'd RB immediately.
> > 
> > Change looks good though and could be useful else where too.
> > 
> > >   }
> > > -static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> > > -				     struct xe_vma_ops *vops)
> > > +static struct dma_fence *vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> > > +						   struct xe_vma_ops *vops)
> > >   {
> > >   	struct drm_exec exec;
> > >   	struct dma_fence *fence;
> > > @@ -2889,7 +2897,6 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> > >   		fence = ops_execute(vm, vops);
> > >   		if (IS_ERR(fence)) {
> > > -			err = PTR_ERR(fence);
> > >   			/* FIXME: Killing VM rather than proper error handling */
> > >   			xe_vm_kill(vm, false);
> > Looks like you are on old baseline before this series landed [1]. I
> > suggest rebasing as those changes creep up in the upper layers a bit.
> > 
> > [1] https://patchwork.freedesktop.org/series/133034/
> 
> Yes, my local tree is from last week. I'll rebase and split out the changes
> to their own patch as suggested.
> 
> > >   			goto unlock;
> > > @@ -2900,7 +2907,7 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,
> > >   unlock:
> > >   	drm_exec_fini(&exec);
> > > -	return err;
> > > +	return fence;
> > >   }
> > >   #define SUPPORTED_FLAGS	\
> > > @@ -3114,6 +3121,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > >   	struct xe_sync_entry *syncs = NULL;
> > >   	struct drm_xe_vm_bind_op *bind_ops;
> > >   	struct xe_vma_ops vops;
> > > +	struct dma_fence *fence;
> > >   	int err;
> > >   	int i;
> > > @@ -3264,7 +3272,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > >   		goto unwind_ops;
> > >   	}
> > > -	err = vm_bind_ioctl_ops_execute(vm, &vops);
> > > +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
> > > +	if (IS_ERR(fence))
> > > +		err = PTR_ERR(fence);
> > > +	else
> > > +		dma_fence_put(fence);
> > >   unwind_ops:
> > >   	if (err && err != -ENODATA)
> > > @@ -3297,6 +3309,80 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > >   	return err;
> > >   }
> > > +/**
> > > + * xe_vm_bind_bo - bind a kernel BO to a VM
> > > + * @vm: VM to bind the BO to
> > > + * @bo: BO to bind
> > > + * @q: exec queue to use for the bind (optional)
> > > + * @addr: address at which to bind the BO
> > > + * @cache_lvl: PAT cache level to use
> > > + *
> > > + * Execute a VM bind map operation on a kernel-owned BO to bind it into a
> > > + * kernel-owned VM.
> > > + *
> > > + * Returns 0 if the ops execution is successful, an errno value otherwise.
> > > + * TODO: return a fence instead.
> > > + */
> > > +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
> > > +				struct xe_exec_queue *q, u64 addr,
> > > +				enum xe_cache_level cache_lvl)
> > > +{
> > > +	struct xe_vma_ops vops;
> > > +	struct drm_gpuva_ops *ops = NULL;
> > > +	struct dma_fence *fence;
> > > +	int err;
> > > +
> > > +	xe_bo_get(bo);
> > > +	xe_vm_get(vm);
> > > +	if (q)
> > > +		xe_exec_queue_get(q);
> > > +
> > > +	down_write(&vm->lock);
> > > +
> > > +	xe_vma_ops_init(&vops, vm, q, NULL, 0);
> > > +
> > > +	ops = vm_bind_ioctl_ops_create(vm, bo, 0, addr, bo->size,
> > > +				       DRM_XE_VM_BIND_OP_MAP, 0,
> > > +				       vm->xe->pat.idx[cache_lvl], 0);
> > > +	if (IS_ERR(ops)) {
> > > +		err = PTR_ERR(ops);
> > > +		goto release_vm_lock;
> > > +	}
> > > +
> > > +	err = vm_bind_ioctl_ops_parse(vm, q, ops, NULL, 0, &vops, true);
> > > +	if (err)
> > > +		goto release_vm_lock;
> > > +
> > > +	/* Nothing to do */
> > > +	if (list_empty(&vops.list)) {
> > Can this ever be true? In the current usage it appear so. Maybe convert
> > to an asset !list_empty to simplify this function slightly?
> 
> will do.
> 
> Daniele
> 
> > 
> > Matt
> > 
> > > +		err = -ENODATA;
> > > +		goto unwind_ops;
> > > +	}
> > > +
> > > +	fence = vm_bind_ioctl_ops_execute(vm, &vops);
> > > +	if (IS_ERR(fence))
> > > +		err = PTR_ERR(fence);
> > > +
> > > +unwind_ops:
> > > +	if (err && err != -ENODATA)
> > > +		vm_bind_ioctl_ops_unwind(vm, &ops, 1);
> > > +
> > > +	drm_gpuva_ops_free(&vm->gpuvm, ops);
> > > +
> > > +release_vm_lock:
> > > +	up_write(&vm->lock);
> > > +
> > > +	if (q)
> > > +		xe_exec_queue_put(q);
> > > +	xe_vm_put(vm);
> > > +	xe_bo_put(bo);
> > > +
> > > +	if (err)
> > > +		fence = ERR_PTR(err);
> > > +
> > > +	return fence;
> > > +}
> > > +
> > >   /**
> > >    * xe_vm_lock() - Lock the vm's dma_resv object
> > >    * @vm: The struct xe_vm whose lock is to be locked
> > > diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
> > > index b481608b12f1..5e298ac90dfc 100644
> > > --- a/drivers/gpu/drm/xe/xe_vm.h
> > > +++ b/drivers/gpu/drm/xe/xe_vm.h
> > > @@ -19,6 +19,8 @@ struct drm_file;
> > >   struct ttm_buffer_object;
> > >   struct ttm_validate_buffer;
> > > +struct dma_fence;
> > > +
> > >   struct xe_exec_queue;
> > >   struct xe_file;
> > >   struct xe_sync_entry;
> > > @@ -248,6 +250,10 @@ int xe_vm_lock_vma(struct drm_exec *exec, struct xe_vma *vma);
> > >   int xe_vm_validate_rebind(struct xe_vm *vm, struct drm_exec *exec,
> > >   			  unsigned int num_fences);
> > > +struct dma_fence *xe_vm_bind_bo(struct xe_vm *vm, struct xe_bo *bo,
> > > +				struct xe_exec_queue *q, u64 addr,
> > > +				enum xe_cache_level cache_lvl);
> > > +
> > >   /**
> > >    * xe_vm_resv() - Return's the vm's reservation object
> > >    * @vm: The vm
> > > diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
> > > index ce1a63a5e3e7..60ce327d303c 100644
> > > --- a/drivers/gpu/drm/xe/xe_vm_types.h
> > > +++ b/drivers/gpu/drm/xe/xe_vm_types.h
> > > @@ -152,6 +152,7 @@ struct xe_vm {
> > >   #define XE_VM_FLAG_BANNED		BIT(5)
> > >   #define XE_VM_FLAG_TILE_ID(flags)	FIELD_GET(GENMASK(7, 6), flags)
> > >   #define XE_VM_FLAG_SET_TILE_ID(tile)	FIELD_PREP(GENMASK(7, 6), (tile)->id)
> > > +#define XE_VM_FLAG_GSC			BIT(8)
> > >   	unsigned long flags;
> > >   	/** @composite_fence_ctx: context composite fence */
> > > -- 
> > > 2.43.0
> > > 
> 

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Souza, Jose]

On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
> PXP prerequisites (SW proxy and HuC auth via GSC) are completed
> asynchronously from driver load, which means that userspace can start
> submitting before we're ready to start a PXP session. Therefore, we need
> a query that userspace can use to check not only if PXP is supported by
> also to wait until the prerequisites are done.
> 
> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_pxp.h   |  1 +
>  drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
>  include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
>  4 files changed, 95 insertions(+)
> 
> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> index e979b47f155a..e39a47aeb050 100644
> --- a/drivers/gpu/drm/xe/xe_pxp.c
> +++ b/drivers/gpu/drm/xe/xe_pxp.c
> @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
>  	return ready;
>  }
>  
> +/**
> + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
> + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
> + *
> + * This function is used for status query from userspace, so the returned value
> + * follow the uapi (see drm_xe_query_pxp_status)
> + *
> + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
> + * is not supported/enabled or if something went wrong in the initialization of
> + * the prerequisites.
> + */
> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
> +{
> +	int ret = 0;
> +
> +	if (!xe_pxp_is_enabled(pxp))
> +		return -ENODEV;
> +
> +	/* if the GSC or HuC FW are in an error state, PXP will never work */
> +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
> +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
> +		return -EIO;
> +
> +	xe_pm_runtime_get(pxp->xe);
> +
> +	/* PXP requires both HuC loaded and GSC proxy initialized */
> +	if (pxp_prerequisites_done(pxp))
> +		ret = 1;
> +
> +	xe_pm_runtime_put(pxp->xe);
> +	return ret;
> +}
> +
>  static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
>  {
>  	struct xe_gt *gt = pxp->gt;
> diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
> index 8f0a3a514fb8..b2aae4abdd0e 100644
> --- a/drivers/gpu/drm/xe/xe_pxp.h
> +++ b/drivers/gpu/drm/xe/xe_pxp.h
> @@ -14,6 +14,7 @@ struct xe_pxp;
>  
>  bool xe_pxp_is_supported(const struct xe_device *xe);
>  bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
>  
>  int xe_pxp_init(struct xe_device *xe);
>  void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
> diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
> index 4e01df6b1b7a..5da9f403c2b9 100644
> --- a/drivers/gpu/drm/xe/xe_query.c
> +++ b/drivers/gpu/drm/xe/xe_query.c
> @@ -22,6 +22,7 @@
>  #include "xe_guc_hwconfig.h"
>  #include "xe_macros.h"
>  #include "xe_mmio.h"
> +#include "xe_pxp.h"
>  #include "xe_ttm_vram_mgr.h"
>  
>  static const u16 xe_to_user_engine_class[] = {
> @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
>  	return ret ? -EFAULT : 0;
>  }
>  
> +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
> +{
> +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
> +	size_t size = sizeof(struct drm_xe_query_pxp_status);
> +	struct drm_xe_query_pxp_status resp;
> +	int ret;
> +
> +	if (query->size == 0) {
> +		query->size = size;
> +		return 0;
> +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
> +		return -EINVAL;
> +	}
> +
> +	if (copy_from_user(&resp, query_ptr, size))
> +		return -EFAULT;
> +
> +	ret = xe_pxp_get_readiness_status(xe->pxp);
> +	if (ret < 0)
> +		return ret;
> +
> +	resp.status = ret;

when status is == 0? does it means that will always succeed? it just matter of time.
For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
not.

> +	resp.supported_session_types =
> +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);

Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...

> +
> +	if (copy_to_user(query_ptr, &resp, size))
> +		 return -EFAULT;
> +
> +	return 0;
> +}
> +
>  static int (* const xe_query_funcs[])(struct xe_device *xe,
>  				      struct drm_xe_device_query *query) = {
>  	query_engines,
> @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
>  	query_engine_cycles,
>  	query_uc_fw_version,
>  	query_oa_units,
> +	query_pxp_status,
>  };
>  
>  int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
> index 631fdc2ed493..746bc16bd220 100644
> --- a/include/uapi/drm/xe_drm.h
> +++ b/include/uapi/drm/xe_drm.h
> @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
>  	__u64 reserved;
>  };
>  
> +/**
> + * struct drm_xe_query_pxp_status - query if PXP is ready
> + *
> + * If PXP is enabled and no fatal error as occurred, the status will be set to
> + * one of the following values:
> + * 0: PXP init still in progress
> + * 1: PXP init complete
> + *
> + * The supported session type bitmask is based on the values in
> + * enum drm_xe_pxp_session_type, including TYPE_NONE.
> + *
> + * If PXP is not enabled or something has gone wrong, the query will be failed
> + * with one of the following error codes:
> + * -ENODEV: PXP not supported or disabled;
> + * -EIO: fatal error occurred during init, so PXP will never be enabled;
> + * -EINVAL: incorrect value provided as part of the query;
> + * -EFAULT: error copying the memory between kernel and userspace.
> + */
> +struct drm_xe_query_pxp_status {
> +	/** @status: current PXP status */
> +	__u32 status;
> +
> +	/** @supported_session_types: bitmask of supported PXP session types */
> +	__u32 supported_session_types;
> +};
> +
>  /**
>   * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
>   * structure to query device information
> @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
>   *    attributes.
>   *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
>   *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
> + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
>   *
>   * If size is set to 0, the driver fills it with the required size for
>   * the requested type of data to query. If size is equal to the required
> @@ -690,6 +717,7 @@ struct drm_xe_device_query {
>  #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
>  #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
>  #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
> +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
>  	/** @query: The type of data to query */
>  	__u32 query;
>  

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Daniele Ceraolo Spurio]

On 7/15/2024 10:54 AM, Souza, Jose wrote:
> On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
>> PXP prerequisites (SW proxy and HuC auth via GSC) are completed
>> asynchronously from driver load, which means that userspace can start
>> submitting before we're ready to start a PXP session. Therefore, we need
>> a query that userspace can use to check not only if PXP is supported by
>> also to wait until the prerequisites are done.
>>
>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>> ---
>>   drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
>>   drivers/gpu/drm/xe/xe_pxp.h   |  1 +
>>   drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
>>   include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
>>   4 files changed, 95 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
>> index e979b47f155a..e39a47aeb050 100644
>> --- a/drivers/gpu/drm/xe/xe_pxp.c
>> +++ b/drivers/gpu/drm/xe/xe_pxp.c
>> @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
>>   	return ready;
>>   }
>>   
>> +/**
>> + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
>> + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
>> + *
>> + * This function is used for status query from userspace, so the returned value
>> + * follow the uapi (see drm_xe_query_pxp_status)
>> + *
>> + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
>> + * is not supported/enabled or if something went wrong in the initialization of
>> + * the prerequisites.
>> + */
>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
>> +{
>> +	int ret = 0;
>> +
>> +	if (!xe_pxp_is_enabled(pxp))
>> +		return -ENODEV;
>> +
>> +	/* if the GSC or HuC FW are in an error state, PXP will never work */
>> +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
>> +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
>> +		return -EIO;
>> +
>> +	xe_pm_runtime_get(pxp->xe);
>> +
>> +	/* PXP requires both HuC loaded and GSC proxy initialized */
>> +	if (pxp_prerequisites_done(pxp))
>> +		ret = 1;
>> +
>> +	xe_pm_runtime_put(pxp->xe);
>> +	return ret;
>> +}
>> +
>>   static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
>>   {
>>   	struct xe_gt *gt = pxp->gt;
>> diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
>> index 8f0a3a514fb8..b2aae4abdd0e 100644
>> --- a/drivers/gpu/drm/xe/xe_pxp.h
>> +++ b/drivers/gpu/drm/xe/xe_pxp.h
>> @@ -14,6 +14,7 @@ struct xe_pxp;
>>   
>>   bool xe_pxp_is_supported(const struct xe_device *xe);
>>   bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
>>   
>>   int xe_pxp_init(struct xe_device *xe);
>>   void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
>> diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
>> index 4e01df6b1b7a..5da9f403c2b9 100644
>> --- a/drivers/gpu/drm/xe/xe_query.c
>> +++ b/drivers/gpu/drm/xe/xe_query.c
>> @@ -22,6 +22,7 @@
>>   #include "xe_guc_hwconfig.h"
>>   #include "xe_macros.h"
>>   #include "xe_mmio.h"
>> +#include "xe_pxp.h"
>>   #include "xe_ttm_vram_mgr.h"
>>   
>>   static const u16 xe_to_user_engine_class[] = {
>> @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
>>   	return ret ? -EFAULT : 0;
>>   }
>>   
>> +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
>> +{
>> +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
>> +	size_t size = sizeof(struct drm_xe_query_pxp_status);
>> +	struct drm_xe_query_pxp_status resp;
>> +	int ret;
>> +
>> +	if (query->size == 0) {
>> +		query->size = size;
>> +		return 0;
>> +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (copy_from_user(&resp, query_ptr, size))
>> +		return -EFAULT;
>> +
>> +	ret = xe_pxp_get_readiness_status(xe->pxp);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	resp.status = ret;
> when status is == 0? does it means that will always succeed? it just matter of time.

no, it just means the init is still in progress. It can still end up in 
an error state if something goes wrong during the init (e.g. GSC proxy 
init failure), although that's unlikely.

> For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
> not.

This query is enough to confirm. That's true for i915 as well, the query 
was added explicitly to avoid having to create a context just to check 
for PXP.

>
>> +	resp.supported_session_types =
>> +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
> Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...

I just added it in to have all values of the enum represented. I can 
take it out no problem if you think it is clearer that way, I'll just 
add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always 
supported.

Daniele

>
>> +
>> +	if (copy_to_user(query_ptr, &resp, size))
>> +		 return -EFAULT;
>> +
>> +	return 0;
>> +}
>> +
>>   static int (* const xe_query_funcs[])(struct xe_device *xe,
>>   				      struct drm_xe_device_query *query) = {
>>   	query_engines,
>> @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
>>   	query_engine_cycles,
>>   	query_uc_fw_version,
>>   	query_oa_units,
>> +	query_pxp_status,
>>   };
>>   
>>   int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>> diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
>> index 631fdc2ed493..746bc16bd220 100644
>> --- a/include/uapi/drm/xe_drm.h
>> +++ b/include/uapi/drm/xe_drm.h
>> @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
>>   	__u64 reserved;
>>   };
>>   
>> +/**
>> + * struct drm_xe_query_pxp_status - query if PXP is ready
>> + *
>> + * If PXP is enabled and no fatal error as occurred, the status will be set to
>> + * one of the following values:
>> + * 0: PXP init still in progress
>> + * 1: PXP init complete
>> + *
>> + * The supported session type bitmask is based on the values in
>> + * enum drm_xe_pxp_session_type, including TYPE_NONE.
>> + *
>> + * If PXP is not enabled or something has gone wrong, the query will be failed
>> + * with one of the following error codes:
>> + * -ENODEV: PXP not supported or disabled;
>> + * -EIO: fatal error occurred during init, so PXP will never be enabled;
>> + * -EINVAL: incorrect value provided as part of the query;
>> + * -EFAULT: error copying the memory between kernel and userspace.
>> + */
>> +struct drm_xe_query_pxp_status {
>> +	/** @status: current PXP status */
>> +	__u32 status;
>> +
>> +	/** @supported_session_types: bitmask of supported PXP session types */
>> +	__u32 supported_session_types;
>> +};
>> +
>>   /**
>>    * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
>>    * structure to query device information
>> @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
>>    *    attributes.
>>    *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
>>    *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
>> + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
>>    *
>>    * If size is set to 0, the driver fills it with the required size for
>>    * the requested type of data to query. If size is equal to the required
>> @@ -690,6 +717,7 @@ struct drm_xe_device_query {
>>   #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
>>   #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
>>   #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
>> +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
>>   	/** @query: The type of data to query */
>>   	__u32 query;
>>   

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Souza, Jose]

On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
> 
> On 7/15/2024 10:54 AM, Souza, Jose wrote:
> > On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
> > > PXP prerequisites (SW proxy and HuC auth via GSC) are completed
> > > asynchronously from driver load, which means that userspace can start
> > > submitting before we're ready to start a PXP session. Therefore, we need
> > > a query that userspace can use to check not only if PXP is supported by
> > > also to wait until the prerequisites are done.
> > > 
> > > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> > > ---
> > >   drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
> > >   drivers/gpu/drm/xe/xe_pxp.h   |  1 +
> > >   drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
> > >   include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
> > >   4 files changed, 95 insertions(+)
> > > 
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> > > index e979b47f155a..e39a47aeb050 100644
> > > --- a/drivers/gpu/drm/xe/xe_pxp.c
> > > +++ b/drivers/gpu/drm/xe/xe_pxp.c
> > > @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
> > >   	return ready;
> > >   }
> > >   
> > > +/**
> > > + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
> > > + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
> > > + *
> > > + * This function is used for status query from userspace, so the returned value
> > > + * follow the uapi (see drm_xe_query_pxp_status)
> > > + *
> > > + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
> > > + * is not supported/enabled or if something went wrong in the initialization of
> > > + * the prerequisites.
> > > + */
> > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
> > > +{
> > > +	int ret = 0;
> > > +
> > > +	if (!xe_pxp_is_enabled(pxp))
> > > +		return -ENODEV;
> > > +
> > > +	/* if the GSC or HuC FW are in an error state, PXP will never work */
> > > +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
> > > +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
> > > +		return -EIO;
> > > +
> > > +	xe_pm_runtime_get(pxp->xe);
> > > +
> > > +	/* PXP requires both HuC loaded and GSC proxy initialized */
> > > +	if (pxp_prerequisites_done(pxp))
> > > +		ret = 1;
> > > +
> > > +	xe_pm_runtime_put(pxp->xe);
> > > +	return ret;
> > > +}
> > > +
> > >   static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
> > >   {
> > >   	struct xe_gt *gt = pxp->gt;
> > > diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
> > > index 8f0a3a514fb8..b2aae4abdd0e 100644
> > > --- a/drivers/gpu/drm/xe/xe_pxp.h
> > > +++ b/drivers/gpu/drm/xe/xe_pxp.h
> > > @@ -14,6 +14,7 @@ struct xe_pxp;
> > >   
> > >   bool xe_pxp_is_supported(const struct xe_device *xe);
> > >   bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
> > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
> > >   
> > >   int xe_pxp_init(struct xe_device *xe);
> > >   void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
> > > diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
> > > index 4e01df6b1b7a..5da9f403c2b9 100644
> > > --- a/drivers/gpu/drm/xe/xe_query.c
> > > +++ b/drivers/gpu/drm/xe/xe_query.c
> > > @@ -22,6 +22,7 @@
> > >   #include "xe_guc_hwconfig.h"
> > >   #include "xe_macros.h"
> > >   #include "xe_mmio.h"
> > > +#include "xe_pxp.h"
> > >   #include "xe_ttm_vram_mgr.h"
> > >   
> > >   static const u16 xe_to_user_engine_class[] = {
> > > @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
> > >   	return ret ? -EFAULT : 0;
> > >   }
> > >   
> > > +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
> > > +{
> > > +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
> > > +	size_t size = sizeof(struct drm_xe_query_pxp_status);
> > > +	struct drm_xe_query_pxp_status resp;
> > > +	int ret;
> > > +
> > > +	if (query->size == 0) {
> > > +		query->size = size;
> > > +		return 0;
> > > +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	if (copy_from_user(&resp, query_ptr, size))
> > > +		return -EFAULT;
> > > +
> > > +	ret = xe_pxp_get_readiness_status(xe->pxp);
> > > +	if (ret < 0)
> > > +		return ret;
> > > +
> > > +	resp.status = ret;
> > when status is == 0? does it means that will always succeed? it just matter of time.
> 
> no, it just means the init is still in progress. It can still end up in 
> an error state if something goes wrong during the init (e.g. GSC proxy 
> init failure), although that's unlikely.
> 
> > For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
> > not.
> 
> This query is enough to confirm. That's true for i915 as well, the query 
> was added explicitly to avoid having to create a context just to check 
> for PXP.

But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
applications if protected stuff is supported or not as one of first steps of Vulkan API.

From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
parameter in the status uAPI?

> 
> > 
> > > +	resp.supported_session_types =
> > > +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
> > Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
> 
> I just added it in to have all values of the enum represented. I can 
> take it out no problem if you think it is clearer that way, I'll just 
> add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always 
> supported.

Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
case were only DRM_XE_PXP_TYPE_NONE is returned.
So I don't think you need to return it as supported.


> 
> Daniele
> 
> > 
> > > +
> > > +	if (copy_to_user(query_ptr, &resp, size))
> > > +		 return -EFAULT;
> > > +
> > > +	return 0;
> > > +}
> > > +
> > >   static int (* const xe_query_funcs[])(struct xe_device *xe,
> > >   				      struct drm_xe_device_query *query) = {
> > >   	query_engines,
> > > @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
> > >   	query_engine_cycles,
> > >   	query_uc_fw_version,
> > >   	query_oa_units,
> > > +	query_pxp_status,
> > >   };
> > >   
> > >   int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > > diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
> > > index 631fdc2ed493..746bc16bd220 100644
> > > --- a/include/uapi/drm/xe_drm.h
> > > +++ b/include/uapi/drm/xe_drm.h
> > > @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
> > >   	__u64 reserved;
> > >   };
> > >   
> > > +/**
> > > + * struct drm_xe_query_pxp_status - query if PXP is ready
> > > + *
> > > + * If PXP is enabled and no fatal error as occurred, the status will be set to
> > > + * one of the following values:
> > > + * 0: PXP init still in progress
> > > + * 1: PXP init complete
> > > + *
> > > + * The supported session type bitmask is based on the values in
> > > + * enum drm_xe_pxp_session_type, including TYPE_NONE.
> > > + *
> > > + * If PXP is not enabled or something has gone wrong, the query will be failed
> > > + * with one of the following error codes:
> > > + * -ENODEV: PXP not supported or disabled;
> > > + * -EIO: fatal error occurred during init, so PXP will never be enabled;
> > > + * -EINVAL: incorrect value provided as part of the query;
> > > + * -EFAULT: error copying the memory between kernel and userspace.
> > > + */
> > > +struct drm_xe_query_pxp_status {
> > > +	/** @status: current PXP status */
> > > +	__u32 status;
> > > +
> > > +	/** @supported_session_types: bitmask of supported PXP session types */
> > > +	__u32 supported_session_types;
> > > +};
> > > +
> > >   /**
> > >    * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
> > >    * structure to query device information
> > > @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
> > >    *    attributes.
> > >    *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
> > >    *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
> > > + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
> > >    *
> > >    * If size is set to 0, the driver fills it with the required size for
> > >    * the requested type of data to query. If size is equal to the required
> > > @@ -690,6 +717,7 @@ struct drm_xe_device_query {
> > >   #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
> > >   #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
> > >   #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
> > > +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
> > >   	/** @query: The type of data to query */
> > >   	__u32 query;
> > >   
> 

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Daniele Ceraolo Spurio]

On 7/15/2024 11:41 AM, Souza, Jose wrote:
> On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
>> On 7/15/2024 10:54 AM, Souza, Jose wrote:
>>> On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
>>>> PXP prerequisites (SW proxy and HuC auth via GSC) are completed
>>>> asynchronously from driver load, which means that userspace can start
>>>> submitting before we're ready to start a PXP session. Therefore, we need
>>>> a query that userspace can use to check not only if PXP is supported by
>>>> also to wait until the prerequisites are done.
>>>>
>>>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>>>> ---
>>>>    drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
>>>>    drivers/gpu/drm/xe/xe_pxp.h   |  1 +
>>>>    drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
>>>>    include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
>>>>    4 files changed, 95 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
>>>> index e979b47f155a..e39a47aeb050 100644
>>>> --- a/drivers/gpu/drm/xe/xe_pxp.c
>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.c
>>>> @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
>>>>    	return ready;
>>>>    }
>>>>    
>>>> +/**
>>>> + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
>>>> + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
>>>> + *
>>>> + * This function is used for status query from userspace, so the returned value
>>>> + * follow the uapi (see drm_xe_query_pxp_status)
>>>> + *
>>>> + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
>>>> + * is not supported/enabled or if something went wrong in the initialization of
>>>> + * the prerequisites.
>>>> + */
>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
>>>> +{
>>>> +	int ret = 0;
>>>> +
>>>> +	if (!xe_pxp_is_enabled(pxp))
>>>> +		return -ENODEV;
>>>> +
>>>> +	/* if the GSC or HuC FW are in an error state, PXP will never work */
>>>> +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
>>>> +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
>>>> +		return -EIO;
>>>> +
>>>> +	xe_pm_runtime_get(pxp->xe);
>>>> +
>>>> +	/* PXP requires both HuC loaded and GSC proxy initialized */
>>>> +	if (pxp_prerequisites_done(pxp))
>>>> +		ret = 1;
>>>> +
>>>> +	xe_pm_runtime_put(pxp->xe);
>>>> +	return ret;
>>>> +}
>>>> +
>>>>    static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
>>>>    {
>>>>    	struct xe_gt *gt = pxp->gt;
>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
>>>> index 8f0a3a514fb8..b2aae4abdd0e 100644
>>>> --- a/drivers/gpu/drm/xe/xe_pxp.h
>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.h
>>>> @@ -14,6 +14,7 @@ struct xe_pxp;
>>>>    
>>>>    bool xe_pxp_is_supported(const struct xe_device *xe);
>>>>    bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
>>>>    
>>>>    int xe_pxp_init(struct xe_device *xe);
>>>>    void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
>>>> diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
>>>> index 4e01df6b1b7a..5da9f403c2b9 100644
>>>> --- a/drivers/gpu/drm/xe/xe_query.c
>>>> +++ b/drivers/gpu/drm/xe/xe_query.c
>>>> @@ -22,6 +22,7 @@
>>>>    #include "xe_guc_hwconfig.h"
>>>>    #include "xe_macros.h"
>>>>    #include "xe_mmio.h"
>>>> +#include "xe_pxp.h"
>>>>    #include "xe_ttm_vram_mgr.h"
>>>>    
>>>>    static const u16 xe_to_user_engine_class[] = {
>>>> @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
>>>>    	return ret ? -EFAULT : 0;
>>>>    }
>>>>    
>>>> +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
>>>> +{
>>>> +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
>>>> +	size_t size = sizeof(struct drm_xe_query_pxp_status);
>>>> +	struct drm_xe_query_pxp_status resp;
>>>> +	int ret;
>>>> +
>>>> +	if (query->size == 0) {
>>>> +		query->size = size;
>>>> +		return 0;
>>>> +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
>>>> +		return -EINVAL;
>>>> +	}
>>>> +
>>>> +	if (copy_from_user(&resp, query_ptr, size))
>>>> +		return -EFAULT;
>>>> +
>>>> +	ret = xe_pxp_get_readiness_status(xe->pxp);
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	resp.status = ret;
>>> when status is == 0? does it means that will always succeed? it just matter of time.
>> no, it just means the init is still in progress. It can still end up in
>> an error state if something goes wrong during the init (e.g. GSC proxy
>> init failure), although that's unlikely.
>>
>>> For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
>>> not.
>> This query is enough to confirm. That's true for i915 as well, the query
>> was added explicitly to avoid having to create a context just to check
>> for PXP.
> But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
> applications if protected stuff is supported or not as one of first steps of Vulkan API.

If you need the answer and the status is 0, you need to wait a bit an 
then check again. There is no other safe way of knowing if PXP fully is 
supported or not.
Note that we expect this to complete relatively quickly after driver 
load; the init itself takes ~500ms, but since it is done in a worker 
thread it can be delayed a bit if the system is busy. The only exception 
is if the mei driver doesn't load, for which case the timeout is 10s, 
but that should in theory never happen unless the HW is in a very bad 
state or the mei driver is totally busted.

>
>  From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
> pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
> parameter in the status uAPI?

I can't block the driver init given that GSC init can be slow and we 
don't want to delay the whole driver load by 500ms or more.

I could add a parameter to block the pxp_status query until the result 
is certain, but is there any reason you can't do such a wait in 
userspace? I think it's better to block in userspace when possible and 
it should be relatively simple to implement, something like:

while (time_ms < timeout_ms) {
         err = ioctl (<query pxp status>);
         if (err || pxp_status == 1)
                 break;

         msleep(50);
         time_ms += 50;
}

I have something similar in the local IGT that I'm using for testing.

Also note that the proposed behavior is basically the same as i915. 
Apart from the defines, the only difference is that in i915 we wait for 
a bit (250ms) on context creation, but if PXP is not ready within that 
time we do return a try again later error. Relevant i915 snippet:

ret = intel_pxp_get_readiness_status(pxp, PXP_READINESS_TIMEOUT);
if (ret < 0) {
         drm_dbg(&pxp->ctrl_gt->i915->drm, "PXP: tried but not-avail 
(%d)", ret);
         return ret;
} else if (ret > 1) {
         return -EIO; /* per UAPI spec, user may retry later */
}

>
>>>> +	resp.supported_session_types =
>>>> +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
>>> Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
>> I just added it in to have all values of the enum represented. I can
>> take it out no problem if you think it is clearer that way, I'll just
>> add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always
>> supported.
> Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
> case were only DRM_XE_PXP_TYPE_NONE is returned.
> So I don't think you need to return it as supported.

ok, I'll update it.

Daniele

>
>
>> Daniele
>>
>>>> +
>>>> +	if (copy_to_user(query_ptr, &resp, size))
>>>> +		 return -EFAULT;
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>>    static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>    				      struct drm_xe_device_query *query) = {
>>>>    	query_engines,
>>>> @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>    	query_engine_cycles,
>>>>    	query_uc_fw_version,
>>>>    	query_oa_units,
>>>> +	query_pxp_status,
>>>>    };
>>>>    
>>>>    int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>>> diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
>>>> index 631fdc2ed493..746bc16bd220 100644
>>>> --- a/include/uapi/drm/xe_drm.h
>>>> +++ b/include/uapi/drm/xe_drm.h
>>>> @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
>>>>    	__u64 reserved;
>>>>    };
>>>>    
>>>> +/**
>>>> + * struct drm_xe_query_pxp_status - query if PXP is ready
>>>> + *
>>>> + * If PXP is enabled and no fatal error as occurred, the status will be set to
>>>> + * one of the following values:
>>>> + * 0: PXP init still in progress
>>>> + * 1: PXP init complete
>>>> + *
>>>> + * The supported session type bitmask is based on the values in
>>>> + * enum drm_xe_pxp_session_type, including TYPE_NONE.
>>>> + *
>>>> + * If PXP is not enabled or something has gone wrong, the query will be failed
>>>> + * with one of the following error codes:
>>>> + * -ENODEV: PXP not supported or disabled;
>>>> + * -EIO: fatal error occurred during init, so PXP will never be enabled;
>>>> + * -EINVAL: incorrect value provided as part of the query;
>>>> + * -EFAULT: error copying the memory between kernel and userspace.
>>>> + */
>>>> +struct drm_xe_query_pxp_status {
>>>> +	/** @status: current PXP status */
>>>> +	__u32 status;
>>>> +
>>>> +	/** @supported_session_types: bitmask of supported PXP session types */
>>>> +	__u32 supported_session_types;
>>>> +};
>>>> +
>>>>    /**
>>>>     * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
>>>>     * structure to query device information
>>>> @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
>>>>     *    attributes.
>>>>     *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
>>>>     *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
>>>> + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
>>>>     *
>>>>     * If size is set to 0, the driver fills it with the required size for
>>>>     * the requested type of data to query. If size is equal to the required
>>>> @@ -690,6 +717,7 @@ struct drm_xe_device_query {
>>>>    #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
>>>>    #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
>>>>    #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
>>>> +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
>>>>    	/** @query: The type of data to query */
>>>>    	__u32 query;
>>>>    

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Souza, Jose]

On Mon, 2024-07-15 at 12:38 -0700, Daniele Ceraolo Spurio wrote:
> 
> On 7/15/2024 11:41 AM, Souza, Jose wrote:
> > On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
> > > On 7/15/2024 10:54 AM, Souza, Jose wrote:
> > > > On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
> > > > > PXP prerequisites (SW proxy and HuC auth via GSC) are completed
> > > > > asynchronously from driver load, which means that userspace can start
> > > > > submitting before we're ready to start a PXP session. Therefore, we need
> > > > > a query that userspace can use to check not only if PXP is supported by
> > > > > also to wait until the prerequisites are done.
> > > > > 
> > > > > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> > > > > ---
> > > > >    drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
> > > > >    drivers/gpu/drm/xe/xe_pxp.h   |  1 +
> > > > >    drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
> > > > >    include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
> > > > >    4 files changed, 95 insertions(+)
> > > > > 
> > > > > diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> > > > > index e979b47f155a..e39a47aeb050 100644
> > > > > --- a/drivers/gpu/drm/xe/xe_pxp.c
> > > > > +++ b/drivers/gpu/drm/xe/xe_pxp.c
> > > > > @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
> > > > >    	return ready;
> > > > >    }
> > > > >    
> > > > > +/**
> > > > > + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
> > > > > + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
> > > > > + *
> > > > > + * This function is used for status query from userspace, so the returned value
> > > > > + * follow the uapi (see drm_xe_query_pxp_status)
> > > > > + *
> > > > > + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
> > > > > + * is not supported/enabled or if something went wrong in the initialization of
> > > > > + * the prerequisites.
> > > > > + */
> > > > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
> > > > > +{
> > > > > +	int ret = 0;
> > > > > +
> > > > > +	if (!xe_pxp_is_enabled(pxp))
> > > > > +		return -ENODEV;
> > > > > +
> > > > > +	/* if the GSC or HuC FW are in an error state, PXP will never work */
> > > > > +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
> > > > > +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
> > > > > +		return -EIO;
> > > > > +
> > > > > +	xe_pm_runtime_get(pxp->xe);
> > > > > +
> > > > > +	/* PXP requires both HuC loaded and GSC proxy initialized */
> > > > > +	if (pxp_prerequisites_done(pxp))
> > > > > +		ret = 1;
> > > > > +
> > > > > +	xe_pm_runtime_put(pxp->xe);
> > > > > +	return ret;
> > > > > +}
> > > > > +
> > > > >    static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
> > > > >    {
> > > > >    	struct xe_gt *gt = pxp->gt;
> > > > > diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
> > > > > index 8f0a3a514fb8..b2aae4abdd0e 100644
> > > > > --- a/drivers/gpu/drm/xe/xe_pxp.h
> > > > > +++ b/drivers/gpu/drm/xe/xe_pxp.h
> > > > > @@ -14,6 +14,7 @@ struct xe_pxp;
> > > > >    
> > > > >    bool xe_pxp_is_supported(const struct xe_device *xe);
> > > > >    bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
> > > > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
> > > > >    
> > > > >    int xe_pxp_init(struct xe_device *xe);
> > > > >    void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
> > > > > diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
> > > > > index 4e01df6b1b7a..5da9f403c2b9 100644
> > > > > --- a/drivers/gpu/drm/xe/xe_query.c
> > > > > +++ b/drivers/gpu/drm/xe/xe_query.c
> > > > > @@ -22,6 +22,7 @@
> > > > >    #include "xe_guc_hwconfig.h"
> > > > >    #include "xe_macros.h"
> > > > >    #include "xe_mmio.h"
> > > > > +#include "xe_pxp.h"
> > > > >    #include "xe_ttm_vram_mgr.h"
> > > > >    
> > > > >    static const u16 xe_to_user_engine_class[] = {
> > > > > @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
> > > > >    	return ret ? -EFAULT : 0;
> > > > >    }
> > > > >    
> > > > > +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
> > > > > +{
> > > > > +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
> > > > > +	size_t size = sizeof(struct drm_xe_query_pxp_status);
> > > > > +	struct drm_xe_query_pxp_status resp;
> > > > > +	int ret;
> > > > > +
> > > > > +	if (query->size == 0) {
> > > > > +		query->size = size;
> > > > > +		return 0;
> > > > > +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
> > > > > +		return -EINVAL;
> > > > > +	}
> > > > > +
> > > > > +	if (copy_from_user(&resp, query_ptr, size))
> > > > > +		return -EFAULT;
> > > > > +
> > > > > +	ret = xe_pxp_get_readiness_status(xe->pxp);
> > > > > +	if (ret < 0)
> > > > > +		return ret;
> > > > > +
> > > > > +	resp.status = ret;
> > > > when status is == 0? does it means that will always succeed? it just matter of time.
> > > no, it just means the init is still in progress. It can still end up in
> > > an error state if something goes wrong during the init (e.g. GSC proxy
> > > init failure), although that's unlikely.
> > > 
> > > > For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
> > > > not.
> > > This query is enough to confirm. That's true for i915 as well, the query
> > > was added explicitly to avoid having to create a context just to check
> > > for PXP.
> > But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
> > applications if protected stuff is supported or not as one of first steps of Vulkan API.
> 
> If you need the answer and the status is 0, you need to wait a bit an 
> then check again. There is no other safe way of knowing if PXP fully is 
> supported or not.
> Note that we expect this to complete relatively quickly after driver 
> load; the init itself takes ~500ms, but since it is done in a worker 
> thread it can be delayed a bit if the system is busy. The only exception 
> is if the mei driver doesn't load, for which case the timeout is 10s, 
> but that should in theory never happen unless the HW is in a very bad 
> state or the mei driver is totally busted.
> 
> > 
> >  From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
> > pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
> > parameter in the status uAPI?
> 
> I can't block the driver init given that GSC init can be slow and we 
> don't want to delay the whole driver load by 500ms or more.

I mean not block driver init but block exec_queue_create or DRM_XE_DEVICE_QUERY_PXP_STATUS.

> 
> I could add a parameter to block the pxp_status query until the result 
> is certain, but is there any reason you can't do such a wait in 
> userspace? I think it's better to block in userspace when possible and 
> it should be relatively simple to implement, something like:
> 
> while (time_ms < timeout_ms) {
>          err = ioctl (<query pxp status>);
>          if (err || pxp_status == 1)
>                  break;
> 
>          msleep(50);
>          time_ms += 50;
> }

The problem of doing this in UMD is that we would need to wait up to 10s in all cases.
KMD could do some checking like if mei is loaded then wait up to 500ms, if not ready after 500ms mark as not supported.

And from what I understood there is no timeout in pxp_prerequisites_done() so even after 10s it would still return 1, so the next application starting
would need to wait again up to 10s.

> 
> I have something similar in the local IGT that I'm using for testing.
> 
> Also note that the proposed behavior is basically the same as i915. 
> Apart from the defines, the only difference is that in i915 we wait for 
> a bit (250ms) on context creation, but if PXP is not ready within that 
> time we do return a try again later error. Relevant i915 snippet:
> 
> ret = intel_pxp_get_readiness_status(pxp, PXP_READINESS_TIMEOUT);
> if (ret < 0) {
>          drm_dbg(&pxp->ctrl_gt->i915->drm, "PXP: tried but not-avail 
> (%d)", ret);
>          return ret;
> } else if (ret > 1) {
>          return -EIO; /* per UAPI spec, user may retry later */
> }
> 
> > 
> > > > > +	resp.supported_session_types =
> > > > > +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
> > > > Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
> > > I just added it in to have all values of the enum represented. I can
> > > take it out no problem if you think it is clearer that way, I'll just
> > > add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always
> > > supported.
> > Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
> > case were only DRM_XE_PXP_TYPE_NONE is returned.
> > So I don't think you need to return it as supported.
> 
> ok, I'll update it.
> 
> Daniele
> 
> > 
> > 
> > > Daniele
> > > 
> > > > > +
> > > > > +	if (copy_to_user(query_ptr, &resp, size))
> > > > > +		 return -EFAULT;
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > >    static int (* const xe_query_funcs[])(struct xe_device *xe,
> > > > >    				      struct drm_xe_device_query *query) = {
> > > > >    	query_engines,
> > > > > @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
> > > > >    	query_engine_cycles,
> > > > >    	query_uc_fw_version,
> > > > >    	query_oa_units,
> > > > > +	query_pxp_status,
> > > > >    };
> > > > >    
> > > > >    int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > > > > diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
> > > > > index 631fdc2ed493..746bc16bd220 100644
> > > > > --- a/include/uapi/drm/xe_drm.h
> > > > > +++ b/include/uapi/drm/xe_drm.h
> > > > > @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
> > > > >    	__u64 reserved;
> > > > >    };
> > > > >    
> > > > > +/**
> > > > > + * struct drm_xe_query_pxp_status - query if PXP is ready
> > > > > + *
> > > > > + * If PXP is enabled and no fatal error as occurred, the status will be set to
> > > > > + * one of the following values:
> > > > > + * 0: PXP init still in progress
> > > > > + * 1: PXP init complete
> > > > > + *
> > > > > + * The supported session type bitmask is based on the values in
> > > > > + * enum drm_xe_pxp_session_type, including TYPE_NONE.
> > > > > + *
> > > > > + * If PXP is not enabled or something has gone wrong, the query will be failed
> > > > > + * with one of the following error codes:
> > > > > + * -ENODEV: PXP not supported or disabled;
> > > > > + * -EIO: fatal error occurred during init, so PXP will never be enabled;
> > > > > + * -EINVAL: incorrect value provided as part of the query;
> > > > > + * -EFAULT: error copying the memory between kernel and userspace.
> > > > > + */
> > > > > +struct drm_xe_query_pxp_status {
> > > > > +	/** @status: current PXP status */
> > > > > +	__u32 status;
> > > > > +
> > > > > +	/** @supported_session_types: bitmask of supported PXP session types */
> > > > > +	__u32 supported_session_types;
> > > > > +};
> > > > > +
> > > > >    /**
> > > > >     * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
> > > > >     * structure to query device information
> > > > > @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
> > > > >     *    attributes.
> > > > >     *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
> > > > >     *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
> > > > > + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
> > > > >     *
> > > > >     * If size is set to 0, the driver fills it with the required size for
> > > > >     * the requested type of data to query. If size is equal to the required
> > > > > @@ -690,6 +717,7 @@ struct drm_xe_device_query {
> > > > >    #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
> > > > >    #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
> > > > >    #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
> > > > > +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
> > > > >    	/** @query: The type of data to query */
> > > > >    	__u32 query;
> > > > >    
> 

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Daniele Ceraolo Spurio]

On 7/15/2024 12:52 PM, Souza, Jose wrote:
> On Mon, 2024-07-15 at 12:38 -0700, Daniele Ceraolo Spurio wrote:
>> On 7/15/2024 11:41 AM, Souza, Jose wrote:
>>> On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
>>>> On 7/15/2024 10:54 AM, Souza, Jose wrote:
>>>>> On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
>>>>>> PXP prerequisites (SW proxy and HuC auth via GSC) are completed
>>>>>> asynchronously from driver load, which means that userspace can start
>>>>>> submitting before we're ready to start a PXP session. Therefore, we need
>>>>>> a query that userspace can use to check not only if PXP is supported by
>>>>>> also to wait until the prerequisites are done.
>>>>>>
>>>>>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>>>>>> ---
>>>>>>     drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
>>>>>>     drivers/gpu/drm/xe/xe_pxp.h   |  1 +
>>>>>>     drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
>>>>>>     include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
>>>>>>     4 files changed, 95 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
>>>>>> index e979b47f155a..e39a47aeb050 100644
>>>>>> --- a/drivers/gpu/drm/xe/xe_pxp.c
>>>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.c
>>>>>> @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
>>>>>>     	return ready;
>>>>>>     }
>>>>>>     
>>>>>> +/**
>>>>>> + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
>>>>>> + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
>>>>>> + *
>>>>>> + * This function is used for status query from userspace, so the returned value
>>>>>> + * follow the uapi (see drm_xe_query_pxp_status)
>>>>>> + *
>>>>>> + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
>>>>>> + * is not supported/enabled or if something went wrong in the initialization of
>>>>>> + * the prerequisites.
>>>>>> + */
>>>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
>>>>>> +{
>>>>>> +	int ret = 0;
>>>>>> +
>>>>>> +	if (!xe_pxp_is_enabled(pxp))
>>>>>> +		return -ENODEV;
>>>>>> +
>>>>>> +	/* if the GSC or HuC FW are in an error state, PXP will never work */
>>>>>> +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
>>>>>> +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
>>>>>> +		return -EIO;
>>>>>> +
>>>>>> +	xe_pm_runtime_get(pxp->xe);
>>>>>> +
>>>>>> +	/* PXP requires both HuC loaded and GSC proxy initialized */
>>>>>> +	if (pxp_prerequisites_done(pxp))
>>>>>> +		ret = 1;
>>>>>> +
>>>>>> +	xe_pm_runtime_put(pxp->xe);
>>>>>> +	return ret;
>>>>>> +}
>>>>>> +
>>>>>>     static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
>>>>>>     {
>>>>>>     	struct xe_gt *gt = pxp->gt;
>>>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
>>>>>> index 8f0a3a514fb8..b2aae4abdd0e 100644
>>>>>> --- a/drivers/gpu/drm/xe/xe_pxp.h
>>>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.h
>>>>>> @@ -14,6 +14,7 @@ struct xe_pxp;
>>>>>>     
>>>>>>     bool xe_pxp_is_supported(const struct xe_device *xe);
>>>>>>     bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
>>>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
>>>>>>     
>>>>>>     int xe_pxp_init(struct xe_device *xe);
>>>>>>     void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
>>>>>> diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
>>>>>> index 4e01df6b1b7a..5da9f403c2b9 100644
>>>>>> --- a/drivers/gpu/drm/xe/xe_query.c
>>>>>> +++ b/drivers/gpu/drm/xe/xe_query.c
>>>>>> @@ -22,6 +22,7 @@
>>>>>>     #include "xe_guc_hwconfig.h"
>>>>>>     #include "xe_macros.h"
>>>>>>     #include "xe_mmio.h"
>>>>>> +#include "xe_pxp.h"
>>>>>>     #include "xe_ttm_vram_mgr.h"
>>>>>>     
>>>>>>     static const u16 xe_to_user_engine_class[] = {
>>>>>> @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
>>>>>>     	return ret ? -EFAULT : 0;
>>>>>>     }
>>>>>>     
>>>>>> +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
>>>>>> +{
>>>>>> +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
>>>>>> +	size_t size = sizeof(struct drm_xe_query_pxp_status);
>>>>>> +	struct drm_xe_query_pxp_status resp;
>>>>>> +	int ret;
>>>>>> +
>>>>>> +	if (query->size == 0) {
>>>>>> +		query->size = size;
>>>>>> +		return 0;
>>>>>> +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
>>>>>> +		return -EINVAL;
>>>>>> +	}
>>>>>> +
>>>>>> +	if (copy_from_user(&resp, query_ptr, size))
>>>>>> +		return -EFAULT;
>>>>>> +
>>>>>> +	ret = xe_pxp_get_readiness_status(xe->pxp);
>>>>>> +	if (ret < 0)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	resp.status = ret;
>>>>> when status is == 0? does it means that will always succeed? it just matter of time.
>>>> no, it just means the init is still in progress. It can still end up in
>>>> an error state if something goes wrong during the init (e.g. GSC proxy
>>>> init failure), although that's unlikely.
>>>>
>>>>> For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
>>>>> not.
>>>> This query is enough to confirm. That's true for i915 as well, the query
>>>> was added explicitly to avoid having to create a context just to check
>>>> for PXP.
>>> But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
>>> applications if protected stuff is supported or not as one of first steps of Vulkan API.
>> If you need the answer and the status is 0, you need to wait a bit an
>> then check again. There is no other safe way of knowing if PXP fully is
>> supported or not.
>> Note that we expect this to complete relatively quickly after driver
>> load; the init itself takes ~500ms, but since it is done in a worker
>> thread it can be delayed a bit if the system is busy. The only exception
>> is if the mei driver doesn't load, for which case the timeout is 10s,
>> but that should in theory never happen unless the HW is in a very bad
>> state or the mei driver is totally busted.
>>
>>>   From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
>>> pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
>>> parameter in the status uAPI?
>> I can't block the driver init given that GSC init can be slow and we
>> don't want to delay the whole driver load by 500ms or more.
> I mean not block driver init but block exec_queue_create or DRM_XE_DEVICE_QUERY_PXP_STATUS.
>
>> I could add a parameter to block the pxp_status query until the result
>> is certain, but is there any reason you can't do such a wait in
>> userspace? I think it's better to block in userspace when possible and
>> it should be relatively simple to implement, something like:
>>
>> while (time_ms < timeout_ms) {
>>           err = ioctl (<query pxp status>);
>>           if (err || pxp_status == 1)
>>                   break;
>>
>>           msleep(50);
>>           time_ms += 50;
>> }
> The problem of doing this in UMD is that we would need to wait up to 10s in all cases.
> KMD could do some checking like if mei is loaded then wait up to 500ms, if not ready after 500ms mark as not supported.

We actually can't do any extra checking inside KMD. I haven't found a 
way of figuring out if mei is working; the mei driver has a number of 
sub-components, one of which is the one we need, and I couldn't find any 
way to figure out if it's not up because we're still waiting for it to 
load or if something went wrong. The only thing we can do is timeout on 
it (more on this below).

>
> And from what I understood there is no timeout in pxp_prerequisites_done() so even after 10s it would still return 1, so the next application starting
> would need to wait again up to 10s.

There is a timeout on the GSC FW init and HuC auth, so the FW status 
will move to error if the init doesn't complete or something goes wrong. 
We check the FW status at the beginning of 
xe_pxp_get_readiness_status(), so  all calls from that point onward 
would return an error code. The status can only be 0 in the first 10s 
from driver load (which is the worst case and should never happen, in 
normal scenario it should resolve within 1s even if there is a problem), 
after that it'll either move to 1 or return an error.

>
>> I have something similar in the local IGT that I'm using for testing.
>>
>> Also note that the proposed behavior is basically the same as i915.
>> Apart from the defines, the only difference is that in i915 we wait for
>> a bit (250ms) on context creation, but if PXP is not ready within that
>> time we do return a try again later error. Relevant i915 snippet:
>>
>> ret = intel_pxp_get_readiness_status(pxp, PXP_READINESS_TIMEOUT);
>> if (ret < 0) {
>>           drm_dbg(&pxp->ctrl_gt->i915->drm, "PXP: tried but not-avail
>> (%d)", ret);
>>           return ret;
>> } else if (ret > 1) {
>>           return -EIO; /* per UAPI spec, user may retry later */
>> }
>>
>>>>>> +	resp.supported_session_types =
>>>>>> +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
>>>>> Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
>>>> I just added it in to have all values of the enum represented. I can
>>>> take it out no problem if you think it is clearer that way, I'll just
>>>> add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always
>>>> supported.
>>> Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
>>> case were only DRM_XE_PXP_TYPE_NONE is returned.
>>> So I don't think you need to return it as supported.
>> ok, I'll update it.
>>
>> Daniele
>>
>>>
>>>> Daniele
>>>>
>>>>>> +
>>>>>> +	if (copy_to_user(query_ptr, &resp, size))
>>>>>> +		 return -EFAULT;
>>>>>> +
>>>>>> +	return 0;
>>>>>> +}
>>>>>> +
>>>>>>     static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>>>     				      struct drm_xe_device_query *query) = {
>>>>>>     	query_engines,
>>>>>> @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>>>     	query_engine_cycles,
>>>>>>     	query_uc_fw_version,
>>>>>>     	query_oa_units,
>>>>>> +	query_pxp_status,
>>>>>>     };
>>>>>>     
>>>>>>     int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>>>>> diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
>>>>>> index 631fdc2ed493..746bc16bd220 100644
>>>>>> --- a/include/uapi/drm/xe_drm.h
>>>>>> +++ b/include/uapi/drm/xe_drm.h
>>>>>> @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
>>>>>>     	__u64 reserved;
>>>>>>     };
>>>>>>     
>>>>>> +/**
>>>>>> + * struct drm_xe_query_pxp_status - query if PXP is ready
>>>>>> + *
>>>>>> + * If PXP is enabled and no fatal error as occurred, the status will be set to
>>>>>> + * one of the following values:
>>>>>> + * 0: PXP init still in progress
>>>>>> + * 1: PXP init complete
>>>>>> + *
>>>>>> + * The supported session type bitmask is based on the values in
>>>>>> + * enum drm_xe_pxp_session_type, including TYPE_NONE.
>>>>>> + *
>>>>>> + * If PXP is not enabled or something has gone wrong, the query will be failed
>>>>>> + * with one of the following error codes:
>>>>>> + * -ENODEV: PXP not supported or disabled;
>>>>>> + * -EIO: fatal error occurred during init, so PXP will never be enabled;
>>>>>> + * -EINVAL: incorrect value provided as part of the query;
>>>>>> + * -EFAULT: error copying the memory between kernel and userspace.
>>>>>> + */
>>>>>> +struct drm_xe_query_pxp_status {
>>>>>> +	/** @status: current PXP status */
>>>>>> +	__u32 status;
>>>>>> +
>>>>>> +	/** @supported_session_types: bitmask of supported PXP session types */
>>>>>> +	__u32 supported_session_types;
>>>>>> +};
>>>>>> +
>>>>>>     /**
>>>>>>      * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
>>>>>>      * structure to query device information
>>>>>> @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
>>>>>>      *    attributes.
>>>>>>      *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
>>>>>>      *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
>>>>>> + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
>>>>>>      *
>>>>>>      * If size is set to 0, the driver fills it with the required size for
>>>>>>      * the requested type of data to query. If size is equal to the required
>>>>>> @@ -690,6 +717,7 @@ struct drm_xe_device_query {
>>>>>>     #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
>>>>>>     #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
>>>>>>     #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
>>>>>> +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
>>>>>>     	/** @query: The type of data to query */
>>>>>>     	__u32 query;
>>>>>>     

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Souza, Jose]

On Mon, 2024-07-15 at 13:06 -0700, Daniele Ceraolo Spurio wrote:
> 
> On 7/15/2024 12:52 PM, Souza, Jose wrote:
> > On Mon, 2024-07-15 at 12:38 -0700, Daniele Ceraolo Spurio wrote:
> > > On 7/15/2024 11:41 AM, Souza, Jose wrote:
> > > > On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
> > > > > On 7/15/2024 10:54 AM, Souza, Jose wrote:
> > > > > > On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
> > > > > > > PXP prerequisites (SW proxy and HuC auth via GSC) are completed
> > > > > > > asynchronously from driver load, which means that userspace can start
> > > > > > > submitting before we're ready to start a PXP session. Therefore, we need
> > > > > > > a query that userspace can use to check not only if PXP is supported by
> > > > > > > also to wait until the prerequisites are done.
> > > > > > > 
> > > > > > > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> > > > > > > ---
> > > > > > >     drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
> > > > > > >     drivers/gpu/drm/xe/xe_pxp.h   |  1 +
> > > > > > >     drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
> > > > > > >     include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
> > > > > > >     4 files changed, 95 insertions(+)
> > > > > > > 
> > > > > > > diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
> > > > > > > index e979b47f155a..e39a47aeb050 100644
> > > > > > > --- a/drivers/gpu/drm/xe/xe_pxp.c
> > > > > > > +++ b/drivers/gpu/drm/xe/xe_pxp.c
> > > > > > > @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
> > > > > > >     	return ready;
> > > > > > >     }
> > > > > > >     
> > > > > > > +/**
> > > > > > > + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
> > > > > > > + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
> > > > > > > + *
> > > > > > > + * This function is used for status query from userspace, so the returned value
> > > > > > > + * follow the uapi (see drm_xe_query_pxp_status)
> > > > > > > + *
> > > > > > > + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
> > > > > > > + * is not supported/enabled or if something went wrong in the initialization of
> > > > > > > + * the prerequisites.
> > > > > > > + */
> > > > > > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
> > > > > > > +{
> > > > > > > +	int ret = 0;
> > > > > > > +
> > > > > > > +	if (!xe_pxp_is_enabled(pxp))
> > > > > > > +		return -ENODEV;
> > > > > > > +
> > > > > > > +	/* if the GSC or HuC FW are in an error state, PXP will never work */
> > > > > > > +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
> > > > > > > +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
> > > > > > > +		return -EIO;
> > > > > > > +
> > > > > > > +	xe_pm_runtime_get(pxp->xe);
> > > > > > > +
> > > > > > > +	/* PXP requires both HuC loaded and GSC proxy initialized */
> > > > > > > +	if (pxp_prerequisites_done(pxp))
> > > > > > > +		ret = 1;
> > > > > > > +
> > > > > > > +	xe_pm_runtime_put(pxp->xe);
> > > > > > > +	return ret;
> > > > > > > +}
> > > > > > > +
> > > > > > >     static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
> > > > > > >     {
> > > > > > >     	struct xe_gt *gt = pxp->gt;
> > > > > > > diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
> > > > > > > index 8f0a3a514fb8..b2aae4abdd0e 100644
> > > > > > > --- a/drivers/gpu/drm/xe/xe_pxp.h
> > > > > > > +++ b/drivers/gpu/drm/xe/xe_pxp.h
> > > > > > > @@ -14,6 +14,7 @@ struct xe_pxp;
> > > > > > >     
> > > > > > >     bool xe_pxp_is_supported(const struct xe_device *xe);
> > > > > > >     bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
> > > > > > > +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
> > > > > > >     
> > > > > > >     int xe_pxp_init(struct xe_device *xe);
> > > > > > >     void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
> > > > > > > diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
> > > > > > > index 4e01df6b1b7a..5da9f403c2b9 100644
> > > > > > > --- a/drivers/gpu/drm/xe/xe_query.c
> > > > > > > +++ b/drivers/gpu/drm/xe/xe_query.c
> > > > > > > @@ -22,6 +22,7 @@
> > > > > > >     #include "xe_guc_hwconfig.h"
> > > > > > >     #include "xe_macros.h"
> > > > > > >     #include "xe_mmio.h"
> > > > > > > +#include "xe_pxp.h"
> > > > > > >     #include "xe_ttm_vram_mgr.h"
> > > > > > >     
> > > > > > >     static const u16 xe_to_user_engine_class[] = {
> > > > > > > @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
> > > > > > >     	return ret ? -EFAULT : 0;
> > > > > > >     }
> > > > > > >     
> > > > > > > +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
> > > > > > > +{
> > > > > > > +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
> > > > > > > +	size_t size = sizeof(struct drm_xe_query_pxp_status);
> > > > > > > +	struct drm_xe_query_pxp_status resp;
> > > > > > > +	int ret;
> > > > > > > +
> > > > > > > +	if (query->size == 0) {
> > > > > > > +		query->size = size;
> > > > > > > +		return 0;
> > > > > > > +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
> > > > > > > +		return -EINVAL;
> > > > > > > +	}
> > > > > > > +
> > > > > > > +	if (copy_from_user(&resp, query_ptr, size))
> > > > > > > +		return -EFAULT;
> > > > > > > +
> > > > > > > +	ret = xe_pxp_get_readiness_status(xe->pxp);
> > > > > > > +	if (ret < 0)
> > > > > > > +		return ret;
> > > > > > > +
> > > > > > > +	resp.status = ret;
> > > > > > when status is == 0? does it means that will always succeed? it just matter of time.
> > > > > no, it just means the init is still in progress. It can still end up in
> > > > > an error state if something goes wrong during the init (e.g. GSC proxy
> > > > > init failure), although that's unlikely.
> > > > > 
> > > > > > For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
> > > > > > not.
> > > > > This query is enough to confirm. That's true for i915 as well, the query
> > > > > was added explicitly to avoid having to create a context just to check
> > > > > for PXP.
> > > > But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
> > > > applications if protected stuff is supported or not as one of first steps of Vulkan API.
> > > If you need the answer and the status is 0, you need to wait a bit an
> > > then check again. There is no other safe way of knowing if PXP fully is
> > > supported or not.
> > > Note that we expect this to complete relatively quickly after driver
> > > load; the init itself takes ~500ms, but since it is done in a worker
> > > thread it can be delayed a bit if the system is busy. The only exception
> > > is if the mei driver doesn't load, for which case the timeout is 10s,
> > > but that should in theory never happen unless the HW is in a very bad
> > > state or the mei driver is totally busted.
> > > 
> > > >   From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
> > > > pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
> > > > parameter in the status uAPI?
> > > I can't block the driver init given that GSC init can be slow and we
> > > don't want to delay the whole driver load by 500ms or more.
> > I mean not block driver init but block exec_queue_create or DRM_XE_DEVICE_QUERY_PXP_STATUS.
> > 
> > > I could add a parameter to block the pxp_status query until the result
> > > is certain, but is there any reason you can't do such a wait in
> > > userspace? I think it's better to block in userspace when possible and
> > > it should be relatively simple to implement, something like:
> > > 
> > > while (time_ms < timeout_ms) {
> > >           err = ioctl (<query pxp status>);
> > >           if (err || pxp_status == 1)
> > >                   break;
> > > 
> > >           msleep(50);
> > >           time_ms += 50;
> > > }
> > The problem of doing this in UMD is that we would need to wait up to 10s in all cases.
> > KMD could do some checking like if mei is loaded then wait up to 500ms, if not ready after 500ms mark as not supported.
> 
> We actually can't do any extra checking inside KMD. I haven't found a 
> way of figuring out if mei is working; the mei driver has a number of 
> sub-components, one of which is the one we need, and I couldn't find any 
> way to figure out if it's not up because we're still waiting for it to 
> load or if something went wrong. The only thing we can do is timeout on 
> it (more on this below).
> 
> > 
> > And from what I understood there is no timeout in pxp_prerequisites_done() so even after 10s it would still return 1, so the next application starting
> > would need to wait again up to 10s.
> 
> There is a timeout on the GSC FW init and HuC auth, so the FW status 
> will move to error if the init doesn't complete or something goes wrong. 
> We check the FW status at the beginning of 
> xe_pxp_get_readiness_status(), so  all calls from that point onward 
> would return an error code. The status can only be 0 in the first 10s 
> from driver load (which is the worst case and should never happen, in 
> normal scenario it should resolve within 1s even if there is a problem), 
> after that it'll either move to 1 or return an error.

Okay will do the wait on UMD but can you add some comments to drm_xe_query_pxp_status with the time expected for PXP to finish initialization?
Mesa reviewers will ask from where did I got the timeout value.

> 
> > 
> > > I have something similar in the local IGT that I'm using for testing.
> > > 
> > > Also note that the proposed behavior is basically the same as i915.
> > > Apart from the defines, the only difference is that in i915 we wait for
> > > a bit (250ms) on context creation, but if PXP is not ready within that
> > > time we do return a try again later error. Relevant i915 snippet:
> > > 
> > > ret = intel_pxp_get_readiness_status(pxp, PXP_READINESS_TIMEOUT);
> > > if (ret < 0) {
> > >           drm_dbg(&pxp->ctrl_gt->i915->drm, "PXP: tried but not-avail
> > > (%d)", ret);
> > >           return ret;
> > > } else if (ret > 1) {
> > >           return -EIO; /* per UAPI spec, user may retry later */
> > > }
> > > 
> > > > > > > +	resp.supported_session_types =
> > > > > > > +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
> > > > > > Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
> > > > > I just added it in to have all values of the enum represented. I can
> > > > > take it out no problem if you think it is clearer that way, I'll just
> > > > > add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always
> > > > > supported.
> > > > Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
> > > > case were only DRM_XE_PXP_TYPE_NONE is returned.
> > > > So I don't think you need to return it as supported.
> > > ok, I'll update it.
> > > 
> > > Daniele
> > > 
> > > > 
> > > > > Daniele
> > > > > 
> > > > > > > +
> > > > > > > +	if (copy_to_user(query_ptr, &resp, size))
> > > > > > > +		 return -EFAULT;
> > > > > > > +
> > > > > > > +	return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > >     static int (* const xe_query_funcs[])(struct xe_device *xe,
> > > > > > >     				      struct drm_xe_device_query *query) = {
> > > > > > >     	query_engines,
> > > > > > > @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
> > > > > > >     	query_engine_cycles,
> > > > > > >     	query_uc_fw_version,
> > > > > > >     	query_oa_units,
> > > > > > > +	query_pxp_status,
> > > > > > >     };
> > > > > > >     
> > > > > > >     int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
> > > > > > > diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
> > > > > > > index 631fdc2ed493..746bc16bd220 100644
> > > > > > > --- a/include/uapi/drm/xe_drm.h
> > > > > > > +++ b/include/uapi/drm/xe_drm.h
> > > > > > > @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
> > > > > > >     	__u64 reserved;
> > > > > > >     };
> > > > > > >     
> > > > > > > +/**
> > > > > > > + * struct drm_xe_query_pxp_status - query if PXP is ready
> > > > > > > + *
> > > > > > > + * If PXP is enabled and no fatal error as occurred, the status will be set to
> > > > > > > + * one of the following values:
> > > > > > > + * 0: PXP init still in progress
> > > > > > > + * 1: PXP init complete
> > > > > > > + *
> > > > > > > + * The supported session type bitmask is based on the values in
> > > > > > > + * enum drm_xe_pxp_session_type, including TYPE_NONE.
> > > > > > > + *
> > > > > > > + * If PXP is not enabled or something has gone wrong, the query will be failed
> > > > > > > + * with one of the following error codes:
> > > > > > > + * -ENODEV: PXP not supported or disabled;
> > > > > > > + * -EIO: fatal error occurred during init, so PXP will never be enabled;
> > > > > > > + * -EINVAL: incorrect value provided as part of the query;
> > > > > > > + * -EFAULT: error copying the memory between kernel and userspace.
> > > > > > > + */
> > > > > > > +struct drm_xe_query_pxp_status {
> > > > > > > +	/** @status: current PXP status */
> > > > > > > +	__u32 status;
> > > > > > > +
> > > > > > > +	/** @supported_session_types: bitmask of supported PXP session types */
> > > > > > > +	__u32 supported_session_types;
> > > > > > > +};
> > > > > > > +
> > > > > > >     /**
> > > > > > >      * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
> > > > > > >      * structure to query device information
> > > > > > > @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
> > > > > > >      *    attributes.
> > > > > > >      *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
> > > > > > >      *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
> > > > > > > + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
> > > > > > >      *
> > > > > > >      * If size is set to 0, the driver fills it with the required size for
> > > > > > >      * the requested type of data to query. If size is equal to the required
> > > > > > > @@ -690,6 +717,7 @@ struct drm_xe_device_query {
> > > > > > >     #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
> > > > > > >     #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
> > > > > > >     #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
> > > > > > > +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
> > > > > > >     	/** @query: The type of data to query */
> > > > > > >     	__u32 query;
> > > > > > >     
> 

Re: [RFC 10/14] drm/xe/pxp: add a query for PXP status

[Daniele Ceraolo Spurio]

On 7/15/2024 1:16 PM, Souza, Jose wrote:
> On Mon, 2024-07-15 at 13:06 -0700, Daniele Ceraolo Spurio wrote:
>> On 7/15/2024 12:52 PM, Souza, Jose wrote:
>>> On Mon, 2024-07-15 at 12:38 -0700, Daniele Ceraolo Spurio wrote:
>>>> On 7/15/2024 11:41 AM, Souza, Jose wrote:
>>>>> On Mon, 2024-07-15 at 11:03 -0700, Daniele Ceraolo Spurio wrote:
>>>>>> On 7/15/2024 10:54 AM, Souza, Jose wrote:
>>>>>>> On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
>>>>>>>> PXP prerequisites (SW proxy and HuC auth via GSC) are completed
>>>>>>>> asynchronously from driver load, which means that userspace can start
>>>>>>>> submitting before we're ready to start a PXP session. Therefore, we need
>>>>>>>> a query that userspace can use to check not only if PXP is supported by
>>>>>>>> also to wait until the prerequisites are done.
>>>>>>>>
>>>>>>>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>>>>>>>> ---
>>>>>>>>      drivers/gpu/drm/xe/xe_pxp.c   | 33 +++++++++++++++++++++++++++++++++
>>>>>>>>      drivers/gpu/drm/xe/xe_pxp.h   |  1 +
>>>>>>>>      drivers/gpu/drm/xe/xe_query.c | 33 +++++++++++++++++++++++++++++++++
>>>>>>>>      include/uapi/drm/xe_drm.h     | 28 ++++++++++++++++++++++++++++
>>>>>>>>      4 files changed, 95 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.c b/drivers/gpu/drm/xe/xe_pxp.c
>>>>>>>> index e979b47f155a..e39a47aeb050 100644
>>>>>>>> --- a/drivers/gpu/drm/xe/xe_pxp.c
>>>>>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.c
>>>>>>>> @@ -60,6 +60,39 @@ static bool pxp_prerequisites_done(const struct xe_pxp *pxp)
>>>>>>>>      	return ready;
>>>>>>>>      }
>>>>>>>>      
>>>>>>>> +/**
>>>>>>>> + * xe_pxp_get_readiness_status - check whether PXP is ready for userspace use
>>>>>>>> + * @pxp: the xe_pxp pointer (can be NULL if PXP is disabled)
>>>>>>>> + *
>>>>>>>> + * This function is used for status query from userspace, so the returned value
>>>>>>>> + * follow the uapi (see drm_xe_query_pxp_status)
>>>>>>>> + *
>>>>>>>> + * Returns: 0 if PXP is not ready yet, 1 if it is ready, an errno value if PXP
>>>>>>>> + * is not supported/enabled or if something went wrong in the initialization of
>>>>>>>> + * the prerequisites.
>>>>>>>> + */
>>>>>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp)
>>>>>>>> +{
>>>>>>>> +	int ret = 0;
>>>>>>>> +
>>>>>>>> +	if (!xe_pxp_is_enabled(pxp))
>>>>>>>> +		return -ENODEV;
>>>>>>>> +
>>>>>>>> +	/* if the GSC or HuC FW are in an error state, PXP will never work */
>>>>>>>> +	if (xe_uc_fw_status_to_error(pxp->gt->uc.huc.fw.status) ||
>>>>>>>> +	    xe_uc_fw_status_to_error(pxp->gt->uc.gsc.fw.status))
>>>>>>>> +		return -EIO;
>>>>>>>> +
>>>>>>>> +	xe_pm_runtime_get(pxp->xe);
>>>>>>>> +
>>>>>>>> +	/* PXP requires both HuC loaded and GSC proxy initialized */
>>>>>>>> +	if (pxp_prerequisites_done(pxp))
>>>>>>>> +		ret = 1;
>>>>>>>> +
>>>>>>>> +	xe_pm_runtime_put(pxp->xe);
>>>>>>>> +	return ret;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>>      static bool pxp_session_is_in_play(struct xe_pxp *pxp, u32 id)
>>>>>>>>      {
>>>>>>>>      	struct xe_gt *gt = pxp->gt;
>>>>>>>> diff --git a/drivers/gpu/drm/xe/xe_pxp.h b/drivers/gpu/drm/xe/xe_pxp.h
>>>>>>>> index 8f0a3a514fb8..b2aae4abdd0e 100644
>>>>>>>> --- a/drivers/gpu/drm/xe/xe_pxp.h
>>>>>>>> +++ b/drivers/gpu/drm/xe/xe_pxp.h
>>>>>>>> @@ -14,6 +14,7 @@ struct xe_pxp;
>>>>>>>>      
>>>>>>>>      bool xe_pxp_is_supported(const struct xe_device *xe);
>>>>>>>>      bool xe_pxp_is_enabled(const struct xe_pxp *pxp);
>>>>>>>> +int xe_pxp_get_readiness_status(struct xe_pxp *pxp);
>>>>>>>>      
>>>>>>>>      int xe_pxp_init(struct xe_device *xe);
>>>>>>>>      void xe_pxp_irq_handler(struct xe_device *xe, u16 iir);
>>>>>>>> diff --git a/drivers/gpu/drm/xe/xe_query.c b/drivers/gpu/drm/xe/xe_query.c
>>>>>>>> index 4e01df6b1b7a..5da9f403c2b9 100644
>>>>>>>> --- a/drivers/gpu/drm/xe/xe_query.c
>>>>>>>> +++ b/drivers/gpu/drm/xe/xe_query.c
>>>>>>>> @@ -22,6 +22,7 @@
>>>>>>>>      #include "xe_guc_hwconfig.h"
>>>>>>>>      #include "xe_macros.h"
>>>>>>>>      #include "xe_mmio.h"
>>>>>>>> +#include "xe_pxp.h"
>>>>>>>>      #include "xe_ttm_vram_mgr.h"
>>>>>>>>      
>>>>>>>>      static const u16 xe_to_user_engine_class[] = {
>>>>>>>> @@ -678,6 +679,37 @@ static int query_oa_units(struct xe_device *xe,
>>>>>>>>      	return ret ? -EFAULT : 0;
>>>>>>>>      }
>>>>>>>>      
>>>>>>>> +static int query_pxp_status(struct xe_device *xe, struct drm_xe_device_query *query)
>>>>>>>> +{
>>>>>>>> +	struct drm_xe_query_pxp_status __user *query_ptr = u64_to_user_ptr(query->data);
>>>>>>>> +	size_t size = sizeof(struct drm_xe_query_pxp_status);
>>>>>>>> +	struct drm_xe_query_pxp_status resp;
>>>>>>>> +	int ret;
>>>>>>>> +
>>>>>>>> +	if (query->size == 0) {
>>>>>>>> +		query->size = size;
>>>>>>>> +		return 0;
>>>>>>>> +	} else if (XE_IOCTL_DBG(xe, query->size != size)) {
>>>>>>>> +		return -EINVAL;
>>>>>>>> +	}
>>>>>>>> +
>>>>>>>> +	if (copy_from_user(&resp, query_ptr, size))
>>>>>>>> +		return -EFAULT;
>>>>>>>> +
>>>>>>>> +	ret = xe_pxp_get_readiness_status(xe->pxp);
>>>>>>>> +	if (ret < 0)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	resp.status = ret;
>>>>>>> when status is == 0? does it means that will always succeed? it just matter of time.
>>>>>> no, it just means the init is still in progress. It can still end up in
>>>>>> an error state if something goes wrong during the init (e.g. GSC proxy
>>>>>> init failure), although that's unlikely.
>>>>>>
>>>>>>> For i915 in some older kernel versions we had to create a gem_context with I915_CONTEXT_PARAM_PROTECTED_CONTENT set to verify for sure if supported or
>>>>>>> not.
>>>>>> This query is enough to confirm. That's true for i915 as well, the query
>>>>>> was added explicitly to avoid having to create a context just to check
>>>>>> for PXP.
>>>>> But as you said above, if status == 0 it could still fail, so UMD will still need create a exec_queue to verify because UMD needs to report to
>>>>> applications if protected stuff is supported or not as one of first steps of Vulkan API.
>>>> If you need the answer and the status is 0, you need to wait a bit an
>>>> then check again. There is no other safe way of knowing if PXP fully is
>>>> supported or not.
>>>> Note that we expect this to complete relatively quickly after driver
>>>> load; the init itself takes ~500ms, but since it is done in a worker
>>>> thread it can be delayed a bit if the system is busy. The only exception
>>>> is if the mei driver doesn't load, for which case the timeout is 10s,
>>>> but that should in theory never happen unless the HW is in a very bad
>>>> state or the mei driver is totally busted.
>>>>
>>>>>    From what I looked to the code, exec_queue will return EBUSY if UMD tries to create a exec_queue with DRM_XE_PXP_TYPE_HWDRM but
>>>>> pxp_prerequisites_done() is still returning false. Can't you make something block in KMD until PXP initialization is running? Maybe add a blocking
>>>>> parameter in the status uAPI?
>>>> I can't block the driver init given that GSC init can be slow and we
>>>> don't want to delay the whole driver load by 500ms or more.
>>> I mean not block driver init but block exec_queue_create or DRM_XE_DEVICE_QUERY_PXP_STATUS.
>>>
>>>> I could add a parameter to block the pxp_status query until the result
>>>> is certain, but is there any reason you can't do such a wait in
>>>> userspace? I think it's better to block in userspace when possible and
>>>> it should be relatively simple to implement, something like:
>>>>
>>>> while (time_ms < timeout_ms) {
>>>>            err = ioctl (<query pxp status>);
>>>>            if (err || pxp_status == 1)
>>>>                    break;
>>>>
>>>>            msleep(50);
>>>>            time_ms += 50;
>>>> }
>>> The problem of doing this in UMD is that we would need to wait up to 10s in all cases.
>>> KMD could do some checking like if mei is loaded then wait up to 500ms, if not ready after 500ms mark as not supported.
>> We actually can't do any extra checking inside KMD. I haven't found a
>> way of figuring out if mei is working; the mei driver has a number of
>> sub-components, one of which is the one we need, and I couldn't find any
>> way to figure out if it's not up because we're still waiting for it to
>> load or if something went wrong. The only thing we can do is timeout on
>> it (more on this below).
>>
>>> And from what I understood there is no timeout in pxp_prerequisites_done() so even after 10s it would still return 1, so the next application starting
>>> would need to wait again up to 10s.
>> There is a timeout on the GSC FW init and HuC auth, so the FW status
>> will move to error if the init doesn't complete or something goes wrong.
>> We check the FW status at the beginning of
>> xe_pxp_get_readiness_status(), so  all calls from that point onward
>> would return an error code. The status can only be 0 in the first 10s
>> from driver load (which is the worst case and should never happen, in
>> normal scenario it should resolve within 1s even if there is a problem),
>> after that it'll either move to 1 or return an error.
> Okay will do the wait on UMD but can you add some comments to drm_xe_query_pxp_status with the time expected for PXP to finish initialization?
> Mesa reviewers will ask from where did I got the timeout value.

Sure. I'll also try make it clearer that the status will move one way or 
the other within that timeout, so follow-up queries won't have to wait.

Daniele

>
>>>> I have something similar in the local IGT that I'm using for testing.
>>>>
>>>> Also note that the proposed behavior is basically the same as i915.
>>>> Apart from the defines, the only difference is that in i915 we wait for
>>>> a bit (250ms) on context creation, but if PXP is not ready within that
>>>> time we do return a try again later error. Relevant i915 snippet:
>>>>
>>>> ret = intel_pxp_get_readiness_status(pxp, PXP_READINESS_TIMEOUT);
>>>> if (ret < 0) {
>>>>            drm_dbg(&pxp->ctrl_gt->i915->drm, "PXP: tried but not-avail
>>>> (%d)", ret);
>>>>            return ret;
>>>> } else if (ret > 1) {
>>>>            return -EIO; /* per UAPI spec, user may retry later */
>>>> }
>>>>
>>>>>>>> +	resp.supported_session_types =
>>>>>>>> +		BIT(DRM_XE_PXP_TYPE_NONE) | BIT(DRM_XE_PXP_TYPE_HWDRM);
>>>>>>> Looks odd to me that you return DRM_XE_PXP_TYPE_NONE as supported...
>>>>>> I just added it in to have all values of the enum represented. I can
>>>>>> take it out no problem if you think it is clearer that way, I'll just
>>>>>> add a note in the interface doc that DRM_XE_PXP_TYPE_NONE is always
>>>>>> supported.
>>>>> Ah just notice that enum drm_xe_pxp_session_type is set in exec_queue. But still, if PXP is not supported/available this will fail, so I can't see a
>>>>> case were only DRM_XE_PXP_TYPE_NONE is returned.
>>>>> So I don't think you need to return it as supported.
>>>> ok, I'll update it.
>>>>
>>>> Daniele
>>>>
>>>>>> Daniele
>>>>>>
>>>>>>>> +
>>>>>>>> +	if (copy_to_user(query_ptr, &resp, size))
>>>>>>>> +		 return -EFAULT;
>>>>>>>> +
>>>>>>>> +	return 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>>      static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>>>>>      				      struct drm_xe_device_query *query) = {
>>>>>>>>      	query_engines,
>>>>>>>> @@ -689,6 +721,7 @@ static int (* const xe_query_funcs[])(struct xe_device *xe,
>>>>>>>>      	query_engine_cycles,
>>>>>>>>      	query_uc_fw_version,
>>>>>>>>      	query_oa_units,
>>>>>>>> +	query_pxp_status,
>>>>>>>>      };
>>>>>>>>      
>>>>>>>>      int xe_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>>>>>>>> diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
>>>>>>>> index 631fdc2ed493..746bc16bd220 100644
>>>>>>>> --- a/include/uapi/drm/xe_drm.h
>>>>>>>> +++ b/include/uapi/drm/xe_drm.h
>>>>>>>> @@ -619,6 +619,32 @@ struct drm_xe_query_uc_fw_version {
>>>>>>>>      	__u64 reserved;
>>>>>>>>      };
>>>>>>>>      
>>>>>>>> +/**
>>>>>>>> + * struct drm_xe_query_pxp_status - query if PXP is ready
>>>>>>>> + *
>>>>>>>> + * If PXP is enabled and no fatal error as occurred, the status will be set to
>>>>>>>> + * one of the following values:
>>>>>>>> + * 0: PXP init still in progress
>>>>>>>> + * 1: PXP init complete
>>>>>>>> + *
>>>>>>>> + * The supported session type bitmask is based on the values in
>>>>>>>> + * enum drm_xe_pxp_session_type, including TYPE_NONE.
>>>>>>>> + *
>>>>>>>> + * If PXP is not enabled or something has gone wrong, the query will be failed
>>>>>>>> + * with one of the following error codes:
>>>>>>>> + * -ENODEV: PXP not supported or disabled;
>>>>>>>> + * -EIO: fatal error occurred during init, so PXP will never be enabled;
>>>>>>>> + * -EINVAL: incorrect value provided as part of the query;
>>>>>>>> + * -EFAULT: error copying the memory between kernel and userspace.
>>>>>>>> + */
>>>>>>>> +struct drm_xe_query_pxp_status {
>>>>>>>> +	/** @status: current PXP status */
>>>>>>>> +	__u32 status;
>>>>>>>> +
>>>>>>>> +	/** @supported_session_types: bitmask of supported PXP session types */
>>>>>>>> +	__u32 supported_session_types;
>>>>>>>> +};
>>>>>>>> +
>>>>>>>>      /**
>>>>>>>>       * struct drm_xe_device_query - Input of &DRM_IOCTL_XE_DEVICE_QUERY - main
>>>>>>>>       * structure to query device information
>>>>>>>> @@ -638,6 +664,7 @@ struct drm_xe_query_uc_fw_version {
>>>>>>>>       *    attributes.
>>>>>>>>       *  - %DRM_XE_DEVICE_QUERY_GT_TOPOLOGY
>>>>>>>>       *  - %DRM_XE_DEVICE_QUERY_ENGINE_CYCLES
>>>>>>>> + *  - %DRM_XE_DEVICE_QUERY_PXP_STATUS
>>>>>>>>       *
>>>>>>>>       * If size is set to 0, the driver fills it with the required size for
>>>>>>>>       * the requested type of data to query. If size is equal to the required
>>>>>>>> @@ -690,6 +717,7 @@ struct drm_xe_device_query {
>>>>>>>>      #define DRM_XE_DEVICE_QUERY_ENGINE_CYCLES	6
>>>>>>>>      #define DRM_XE_DEVICE_QUERY_UC_FW_VERSION	7
>>>>>>>>      #define DRM_XE_DEVICE_QUERY_OA_UNITS		8
>>>>>>>> +#define DRM_XE_DEVICE_QUERY_PXP_STATUS		9
>>>>>>>>      	/** @query: The type of data to query */
>>>>>>>>      	__u32 query;
>>>>>>>>      

Re: [RFC 00/14] Add PXP HWDRM support

[Souza, Jose]

On Fri, 2024-07-12 at 14:28 -0700, Daniele Ceraolo Spurio wrote:
> PXP (Protected Xe Path) allows execution and flip to display of protected
> (i.e. encrypted) objects. The HW supports multiple types of PXP, but
> this series only introduces support for PXP HWDRM, which is mainly
> targeted at encrypting data that is going to be displayed.
> 
> Even though we only plan to support 1 type of PXP for now, the interface
> has been designed to allow support for other PXP types to be added at a
> later point in time.
> 
> A user is expected to mark both BO and exec_queues as using PXP and the
> driver will make sure that PXP is running and that the encryption is
> valid and that no execution happens with an outdated encryption.
> 
> RFC: The series is in a decent state and all my local tests are working
> on MTL, but I'm seeing failures on LNL; those are currently trending to
> being issues with the contents of the test batch, but I didn't want to
> hold the series back until I was sure of that because there are a couple
> of choices I made where I'd like feedback, so sending it as a RFC so I
> can get that feedback while I complete the debug. The 2 areas in which
> I'd like feedback are:
> 
> 1) VM/BO changes:
>     - PXP requires PPGTT submissions from inside the driver, which isn't
>       currently supported. I've hooked it up (patch 3), but I'd like an
>       expert eye to confirm I didn't do anything stupid.
>     - We need to reject submissions using VMs with mapped invalid BOs.
>       I've done this by checking all mapped BOs for PXP submissions
>       (patch 11), but I'm open to ideas on how to do this differently.
> 
> 2) API: as mentioned above, I've kept this flexible. Just wanted to
>    confirm no one had anything against the approach. See patches 9-11. 

Other then the comment in patch 10 uAPI looks good.

> 
> Cc: José Roberto de Souza <jose.souza@intel.com>
> Cc: Alan Previn <alan.previn.teres.alexis@intel.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> 
> Daniele Ceraolo Spurio (14):
>   drm/xe: Make exec_queue_kill safe to call twice
>   drm/xe/pxp: Initialize PXP structure and KCR reg
>   drm/xe/pxp: Allocate PXP execution resources
>   drm/xe/pxp: Add VCS inline termination support
>   drm/xe/pxp: Add GSC session invalidation support
>   drm/xe/pxp: Handle the PXP termination interrupt
>   drm/xe/pxp: Add GSC session initialization support
>   drm/xe/pxp: Add hooks to mark an exec queue as using PXP
>   drm/xe/pxp: Add API to mark a queue as using PXP
>   drm/xe/pxp: add a query for PXP status
>   drm/xe/pxp: Add API to mark a BO as using PXP
>   drm/xe/pxp: add PXP PM support
>   drm/xe/pxp: Add PXP debugfs support
>   drm/xe/pxp: Enable PXP for MTL
> 
>  drivers/gpu/drm/xe/Makefile                   |   3 +
>  drivers/gpu/drm/xe/abi/gsc_pxp_commands_abi.h |  40 +
>  .../xe/compat-i915-headers/pxp/intel_pxp.h    |  14 +-
>  .../gpu/drm/xe/instructions/xe_instr_defs.h   |   1 +
>  .../gpu/drm/xe/instructions/xe_mfx_commands.h |  29 +
>  .../gpu/drm/xe/instructions/xe_mi_commands.h  |   5 +
>  drivers/gpu/drm/xe/regs/xe_engine_regs.h      |   1 +
>  drivers/gpu/drm/xe/regs/xe_gt_regs.h          |   8 +
>  drivers/gpu/drm/xe/regs/xe_pxp_regs.h         |  23 +
>  drivers/gpu/drm/xe/xe_bo.c                    | 100 ++-
>  drivers/gpu/drm/xe/xe_bo.h                    |   5 +
>  drivers/gpu/drm/xe/xe_bo_types.h              |   3 +
>  drivers/gpu/drm/xe/xe_debugfs.c               |   3 +
>  drivers/gpu/drm/xe/xe_device.c                |   6 +
>  drivers/gpu/drm/xe/xe_device_types.h          |   8 +-
>  drivers/gpu/drm/xe/xe_exec.c                  |   6 +
>  drivers/gpu/drm/xe/xe_exec_queue.c            |  48 +-
>  drivers/gpu/drm/xe/xe_exec_queue.h            |   5 +
>  drivers/gpu/drm/xe/xe_exec_queue_types.h      |   7 +
>  drivers/gpu/drm/xe/xe_hw_engine.c             |   2 +-
>  drivers/gpu/drm/xe/xe_irq.c                   |  20 +-
>  drivers/gpu/drm/xe/xe_lrc.c                   |  14 +-
>  drivers/gpu/drm/xe/xe_lrc.h                   |   7 +-
>  drivers/gpu/drm/xe/xe_pci.c                   |   3 +
>  drivers/gpu/drm/xe/xe_pm.c                    |  42 +-
>  drivers/gpu/drm/xe/xe_pxp.c                   | 713 ++++++++++++++++++
>  drivers/gpu/drm/xe/xe_pxp.h                   |  32 +
>  drivers/gpu/drm/xe/xe_pxp_debugfs.c           | 120 +++
>  drivers/gpu/drm/xe/xe_pxp_debugfs.h           |  13 +
>  drivers/gpu/drm/xe/xe_pxp_submit.c            | 549 ++++++++++++++
>  drivers/gpu/drm/xe/xe_pxp_submit.h            |  20 +
>  drivers/gpu/drm/xe/xe_pxp_types.h             | 103 +++
>  drivers/gpu/drm/xe/xe_query.c                 |  33 +
>  drivers/gpu/drm/xe/xe_ring_ops.c              |   4 +-
>  drivers/gpu/drm/xe/xe_vm.c                    | 148 +++-
>  drivers/gpu/drm/xe/xe_vm.h                    |   8 +
>  drivers/gpu/drm/xe/xe_vm_types.h              |   1 +
>  include/uapi/drm/xe_drm.h                     |  83 +-
>  38 files changed, 2190 insertions(+), 40 deletions(-)
>  create mode 100644 drivers/gpu/drm/xe/instructions/xe_mfx_commands.h
>  create mode 100644 drivers/gpu/drm/xe/regs/xe_pxp_regs.h
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp.c
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp.h
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.c
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_debugfs.h
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.c
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_submit.h
>  create mode 100644 drivers/gpu/drm/xe/xe_pxp_types.h
>