[PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[Nareshkumar Gollakoti]

This series adds initial Shared Virtual Addressing (SVA) support for xe
on platforms where ATS and PASID are available but PRI is not enabled or
not supported.

For SVA, the device must be able to issue memory transactions tagged
with a PASID so that address translation is performed in the process
address space rather than in a driver owned address space. This series
adds the xe side infrastructure needed for that flow: VM support for SVA
PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID
for ATS and PASID enabled contexts.

On systems with PRI, device page faults can be surfaced and resolved
through the hardware page request interface, so the driver does not need
to implement a separate fault servicing path for SVA.
On non PRI systems, that hardware assisted fault path is unavailable.In
that case, the driver still needs a way to resolve faults against the
backing process address space so execution can proceed.

To support that case, this series introduces a CPU mirror VMA and uses
it as part of a driver managed SVM path for SVA faults without PRI.The
purpose of the CPU mirror VMA is to provide a CPU side VMA
representation for the address range being accesses through SVA,so the
driver can rely on the kernel MM/VMA machinery to validate mappings and
resolve backing pages for fault handling in the absence of PRI.In
other words, since the device cannot issue PRI requests, the driver
needs a CPU visible VMA context that mirrors the GPU SVA range closely
enough to let faults be service through normal MM paths.

At a high level, this series does the following:

 1. log ATS/PASID enablement state during probe,
 2. add VM infrastructure needed for SVA PASID support,
 3. add a helper to retrieve the VM SVA PASID,
 4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
 5. add a PCI helper for ATS/PASID operation without PRI,
 6. add a CPU mirror VMA for handling SVA faults on non PRI platforms

Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
this flow and have been validated internally.Patch 6 adds the non PRI
fault servicing path, and feedback on that overall approach would be
especially appreciated.

Himal Prasad Ghimiray (1):
  drm/xe/vm: Setup SVA infrastructure for PASID support

Nareshkumar Gollakoti (5):
  drm/xe/pci: log ATS/PASID enablement state at probe
  drm/xe/vm: add helper to retrieve VM SVA PASID
  drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
  drm/xe/pci: add helper for ATS/PASID without PRI
  drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
 drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
 drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
 drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
 drivers/gpu/drm/xe/xe_vm.c              | 56 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h              |  2 +
 drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
 9 files changed, 184 insertions(+)

-- 
2.43.0

[PATCH 1/6] drm/xe/pci: log ATS/PASID enablement state at probe

[Nareshkumar Gollakoti]

Add debug logging in xe_pci_probe() for ATS and PASID enablement state
when the corresponding capabilities are present.
This makes platform and IOMMU related bring up issues easier to diagnose.

Also add a small helper to report weather ATS and PASID are both enabled,
for use follow up checks that depend on both features.

Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/xe_pci.c | 7 +++++++
 drivers/gpu/drm/xe/xe_pci.h | 6 ++++++
 2 files changed, 13 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c
index e44b74b9aa27..40c0f4c42ed3 100644
--- a/drivers/gpu/drm/xe/xe_pci.c
+++ b/drivers/gpu/drm/xe/xe_pci.c
@@ -1275,6 +1275,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des
 		xe_step_name(xe->info.step.media),
 		xe_step_name(xe->info.step.basedie));
 
+	if (pdev->ats_cap)
+		drm_dbg(&xe->drm, "Arch ATS = %s",
+			str_enabled_disabled(pdev->ats_enabled));
+	if (pdev->pasid_cap)
+		drm_dbg(&xe->drm, "Arch PASID = %s",
+			str_enabled_disabled(pdev->pasid_enabled));
+
 	drm_dbg(&xe->drm, "SR-IOV support: %s (mode: %s)\n",
 		str_yes_no(xe_device_has_sriov(xe)),
 		xe_sriov_mode_to_string(xe_device_sriov_mode(xe)));
diff --git a/drivers/gpu/drm/xe/xe_pci.h b/drivers/gpu/drm/xe/xe_pci.h
index 11bcc5fe2c5b..5f784229a5ec 100644
--- a/drivers/gpu/drm/xe/xe_pci.h
+++ b/drivers/gpu/drm/xe/xe_pci.h
@@ -5,6 +5,7 @@
 
 #ifndef _XE_PCI_H_
 #define _XE_PCI_H_
+#include <linux/pci.h>
 
 struct pci_dev;
 
@@ -12,4 +13,9 @@ int xe_register_pci_driver(void);
 void xe_unregister_pci_driver(void);
 struct xe_device *xe_pci_to_pf_device(struct pci_dev *pdev);
 
+static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
+{
+	return pdev->ats_enabled && pdev->pasid_enabled;
+}
+
 #endif
-- 
2.43.0

Re: [PATCH 1/6] drm/xe/pci: log ATS/PASID enablement state at probe

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] drm/xe/pci: xe_pci_ats_pasid_enabled() always returns false on SR-IOV VFs
- [Low] drm/xe/pci: build failure in __xe_pci_probe() without CONFIG_PCI_ATS/PASID
--

commit 9613705bbc43ffcbfae8d92b0e6cff920caffa87
Author: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>

drm/xe/pci: log ATS/PASID enablement state at probe

This commit logs ATS and PASID enablement states during __xe_pci_probe() when
their PCI capabilities are present, and introduces xe_pci_ats_pasid_enabled()
to check whether both features are enabled on a PCI device.

> diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c
> index 44587b5f4df60..7c8a4286b657a 100644
> --- a/drivers/gpu/drm/xe/xe_pci.c
> +++ b/drivers/gpu/drm/xe/xe_pci.c
> @@ -1281,6 +1281,13 @@ static int __xe_pci_probe(struct pci_dev *pdev, const struct xe_device_desc *des
>  		xe_step_name(xe->info.step.media),
>  		xe_step_name(xe->info.step.basedie));
>  
> +	if (pdev->ats_cap)
> +		drm_dbg(&xe->drm, "Arch ATS = %s",
> +			str_enabled_disabled(pdev->ats_enabled));
> +	if (pdev->pasid_cap)
> +		drm_dbg(&xe->drm, "Arch PASID = %s",
> +			str_enabled_disabled(pdev->pasid_enabled));

[Severity: Low]
Will __xe_pci_probe() in drivers/gpu/drm/xe/xe_pci.c fail to build when
CONFIG_PCI_ATS or CONFIG_PCI_PASID is disabled?

In include/linux/pci.h, struct pci_dev only defines ats_cap and pasid_cap
when their respective Kconfig options are enabled:

#ifdef CONFIG_PCI_ATS
	...
	u16		ats_cap;	/* ATS Capability offset */
	u8		ats_stu;	/* ATS Smallest Translation Unit */
#endif
...
#ifdef CONFIG_PCI_PASID
	u16		pasid_cap;	/* PASID Capability offset */
	u16		pasid_features;
#endif

Since CONFIG_DRM_XE in drivers/gpu/drm/xe/Kconfig only depends on DRM and
PCI without selecting or depending on PCI_ATS or PCI_PASID (for example, in
drivers/gpu/drm/xe/.kunitconfig), accessing pdev->ats_cap and pdev->pasid_cap
directly without IS_ENABLED() guards breaks the build when either option is
not set.

> +
>  	drm_dbg(&xe->drm, "SR-IOV support: %s (mode: %s)\n",
>  		str_yes_no(xe_device_has_sriov(xe)),
>  		xe_sriov_mode_to_string(xe_device_sriov_mode(xe)));
> diff --git a/drivers/gpu/drm/xe/xe_pci.h b/drivers/gpu/drm/xe/xe_pci.h
> index 11bcc5fe2c5b9..5f784229a5ecf 100644
> --- a/drivers/gpu/drm/xe/xe_pci.h
> +++ b/drivers/gpu/drm/xe/xe_pci.h
> @@ -5,6 +5,7 @@
>  
>  #ifndef _XE_PCI_H_
>  #define _XE_PCI_H_
> +#include <linux/pci.h>
>  
>  struct pci_dev;
>  
> @@ -12,4 +13,9 @@ int xe_register_pci_driver(void);
>  void xe_unregister_pci_driver(void);
>  struct xe_device *xe_pci_to_pf_device(struct pci_dev *pdev);
>  
> +static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
> +{
> +	return pdev->ats_enabled && pdev->pasid_enabled;
> +}

[Severity: Medium]
How does xe_pci_ats_pasid_enabled() in drivers/gpu/drm/xe/xe_pci.h (as well
as the PASID check in __xe_pci_probe()) behave on SR-IOV Virtual Functions?

In drivers/pci/ats.c, pci_enable_pasid() notes that VFs do not implement
their own PASID capability (so pdev->pasid_cap is 0) and instead share the
PF's PASID configuration:

	if (pdev->is_virtfn) {
		if (pci_physfn(pdev)->pasid_enabled)
			return 0;
		return -EINVAL;
	}

When the IOMMU enables PASID on a VF via pci_enable_pasid(), it checks
pci_physfn(pdev)->pasid_enabled and returns 0 without setting
pdev->pasid_enabled on the VF's pci_dev (whereas pci_enable_ats() sets
pdev->ats_enabled on the VF).

Because xe_pci_ats_pasid_enabled() and __xe_pci_probe() inspect
pdev->pasid_enabled and pdev->pasid_cap directly instead of checking
pci_physfn(pdev) (or calling pci_pasid_status(pdev)), won't
xe_pci_ats_pasid_enabled() always return false on VFs and __xe_pci_probe()
always skip logging the PASID state on VFs?

> +
>  #endif

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261021041117.1774877-1-naresh.kumar.g@intel.com?part=1

[PATCH 2/6] drm/xe/vm: Setup SVA infrastructure for PASID support

[Nareshkumar Gollakoti]

From: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>

Add sva_handle to struct xe_vm to represent the device-mm bond
established via iommu_sva_bind_device(). This handle will be used to
retrieve the PASID for SVM-enabled VMs

Introduce xe_vm_sva_bind() to bind the VM to the current process
address space using Shared Virtual Addressing(SVA) when ATS and
PASID are enabled on the device. This allows the GPU to shate the
same virtual address space as the CPU.

If sva binding fails, the VM continues without PASID support.
Also introduce a cleanup helper to unbind during VM teardown.

Signed-off-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c       | 40 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm_types.h |  6 +++++
 2 files changed, 46 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 761bc136145f..b1d37e5a673f 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -45,6 +45,7 @@
 #include "xe_trace_bo.h"
 #include "xe_vm_madvise.h"
 #include "xe_wa.h"
+#include "xe_pci.h"
 
 static struct drm_gem_object *xe_vm_obj(struct xe_vm *vm)
 {
@@ -1648,6 +1649,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
 	up_read(&vm->exec_queues.lock);
 }
 
+static void xe_vm_sva_handle_init(struct xe_vm *vm)
+{
+	struct device *dev = vm->xe->drm.dev;
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct iommu_sva *sva_handle;
+
+	/*
+	 * Assumes the IOMMU driver has already decided and latched the
+	 * device's ATS/PASID enablement by the time any VM is created (true
+	 * for Intel VT-d, which enables both eagerly at IOMMU probe). If a
+	 * backend instead enables them lazily on first SVA bind, this check
+	 * will never see them set and SVA/PASID support is silently skipped.
+	 */
+	if (!xe_pci_ats_pasid_enabled(pdev))
+		return;
+
+	sva_handle = iommu_sva_bind_device(dev, current->mm);
+	if (IS_ERR(sva_handle)) {
+		drm_err(&vm->xe->drm, "SVA binding failed, preventing PASID creation.\n");
+		return;
+	}
+
+	vm->sva_handle = sva_handle;
+}
+
+static void xe_vm_sva_handle_fini(struct xe_vm *vm)
+{
+	xe_assert(vm->xe, xe_vm_is_closed(vm));
+	if (!vm->sva_handle)
+		return;
+
+	iommu_sva_unbind_device(vm->sva_handle);
+	vm->sva_handle = NULL;
+}
+
 struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)
 {
 	struct drm_gem_object *vm_resv_obj;
@@ -1994,6 +2030,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 
 	xe_svm_fini(vm);
 
+	xe_vm_sva_handle_fini(vm);
 	up_write(&vm->lock);
 
 	down_write(&xe->usm.lock);
@@ -2159,6 +2196,9 @@ int xe_vm_create_ioctl(struct drm_device *dev, void *data,
 	if (IS_ERR(vm))
 		return PTR_ERR(vm);
 
+	if (flags & XE_VM_FLAG_FAULT_MODE)
+		xe_vm_sva_handle_init(vm);
+
 #if IS_ENABLED(CONFIG_DRM_XE_DEBUG_MEM)
 	/* Warning: Security issue - never enable by default */
 	args->reserved[0] = xe_bo_main_addr(vm->pt_root[0]->bo, XE_PAGE_SIZE);
diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
index fb9305124679..113807fa4ea0 100644
--- a/drivers/gpu/drm/xe/xe_vm_types.h
+++ b/drivers/gpu/drm/xe/xe_vm_types.h
@@ -11,6 +11,7 @@
 #include <drm/drm_pagemap_util.h>
 
 #include <linux/dma-resv.h>
+#include <linux/iommu.h>
 #include <linux/kref.h>
 #include <linux/mmu_notifier.h>
 #include <linux/scatterlist.h>
@@ -423,6 +424,11 @@ struct xe_vm {
 	bool batch_invalidate_tlb;
 	/** @xef: Xe file handle for tracking this VM's drm client */
 	struct xe_file *xef;
+	/**
+	 * @sva_handle: the sva_handle to a device-mm bond, populated using
+	 * iommu_sva_bind_device(). To be used to determine pasid in case of svm vm.
+	 */
+	struct iommu_sva *sva_handle;
 };
 
 /** struct xe_vma_op_map - VMA map operation */
-- 
2.43.0

Re: [PATCH 2/6] drm/xe/vm: Setup SVA infrastructure for PASID support

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] drm/xe/vm: missing SVA unbind on device removal leaks DMA owner and crashes
--

commit 08c0c9b32bbf8f35bf925eec1c7c7d7bb127ed1a
Author: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>

drm/xe/vm: Setup SVA infrastructure for PASID support

This commit adds an sva_handle field to struct xe_vm and introduces helpers to
bind and unbind a fault-mode VM to the current process address space via
iommu_sva_bind_device() and iommu_sva_unbind_device().

> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index b98f2e56bf6f4..3504b890ababf 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c

[ ... ]

> @@ -1674,6 +1675,41 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
>  	up_read(&vm->exec_queues.lock);
>  }
>  
> +static void xe_vm_sva_handle_init(struct xe_vm *vm)
> +{
> +	struct device *dev = vm->xe->drm.dev;
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct iommu_sva *sva_handle;
> +
> +	/*
> +	 * Assumes the IOMMU driver has already decided and latched the
> +	 * device's ATS/PASID enablement by the time any VM is created (true
> +	 * for Intel VT-d, which enables both eagerly at IOMMU probe). If a
> +	 * backend instead enables them lazily on first SVA bind, this check
> +	 * will never see them set and SVA/PASID support is silently skipped.
> +	 */
> +	if (!xe_pci_ats_pasid_enabled(pdev))
> +		return;
> +
> +	sva_handle = iommu_sva_bind_device(dev, current->mm);
> +	if (IS_ERR(sva_handle)) {
> +		drm_err(&vm->xe->drm, "SVA binding failed, preventing PASID creation.\n");
> +		return;
> +	}
> +
> +	vm->sva_handle = sva_handle;
> +}
> +
> +static void xe_vm_sva_handle_fini(struct xe_vm *vm)
> +{
> +	xe_assert(vm->xe, xe_vm_is_closed(vm));
> +	if (!vm->sva_handle)
> +		return;
> +
> +	iommu_sva_unbind_device(vm->sva_handle);
> +	vm->sva_handle = NULL;
> +}

[Severity: High]
What happens if the pci device is unbound from the driver or hot-unplugged
while a userspace process still holds an open fault-mode xe_vm?

When xe_vm_create_ioctl() calls xe_vm_sva_handle_init(),
iommu_sva_bind_device() attaches a pasid domain and inserts an entry into
dev->iommu_group->pasid_array. However, vm->sva_handle is only unbound in
xe_vm_sva_handle_fini() during xe_vm_close_and_put(), which is tied to the
userspace file descriptor and vm lifetime rather than device removal in
drivers/gpu/drm/xe/xe_device.c:

void xe_device_remove(struct xe_device *xe)
{
	xe_display_unregister(xe);

	drm_dev_unplug(&xe->drm);

	xe_bo_pci_dev_remove_all(xe);
}

In addition, xe_vm_sva_handle_init() does not guard the
iommu_sva_bind_device() call with drm_dev_enter(), so an in-flight
xe_vm_create_ioctl() can also establish a new bond after drm_dev_unplug()
completes.

When __device_release_driver() finishes xe_pci_remove() -> xe_device_remove(),
it immediately calls pci_dma_cleanup() -> iommu_device_unuse_default_domain()
in drivers/iommu/iommu.c:

	mutex_lock(&group->mutex);
	if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
		group->owner_cnt--;

	mutex_unlock(&group->mutex);

Because group->pasid_array is not empty while vm->sva_handle remains bound,
does this trigger the WARN_ON() and leak group->owner_cnt?

Furthermore, if the pci device is hot-unplugged and removed from the bus,
iommu_release_device() -> __iommu_group_remove_device() ->
iommu_deinit_device() clears dev->iommu_group to NULL. When userspace later
closes the file descriptor or destroys the vm:

xe_vm_close_and_put()
  xe_vm_sva_handle_fini()
    iommu_sva_unbind_device()
      iommu_detach_device_pasid()

drivers/iommu/iommu.c:iommu_detach_device_pasid() requires the caller to still
be a probed driver on dev and dereferences dev->iommu_group directly:

void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev,
			       ioasid_t pasid)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	mutex_lock(&group->mutex);

Wouldn't calling iommu_sva_unbind_device() after device removal dereference a
NULL group pointer here? Should xe_device_remove() unbind active sva_handle
bonds before driver detach completes, and should xe_vm_sva_handle_init()
protect the bind with drm_dev_enter() / drm_dev_exit()?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261021041117.1774877-1-naresh.kumar.g@intel.com?part=2

[PATCH 3/6] drm/xe/vm: add helper to retrieve VM SVA PASID

[Nareshkumar Gollakoti]

Add xe_vm_get_sva_pasid() to return the PASID associated with a VM SVA
binding.

The helper checks whether the VM has a valid SVA handle and, if so,
returns the PASID obtained from iommu_sva_get_pasid(). If no SVA handle
is present, return -EINVAL.

Export the helper in xe_vm.h so other XE components can query the VM's
PASID when programming per-context state.

Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index b1d37e5a673f..dcaeeeb94721 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1649,6 +1649,22 @@ static void xe_vm_init_prove_locking(struct xe_device *xe, struct xe_vm *vm)
 	up_read(&vm->exec_queues.lock);
 }
 
+/**
+ * xe_vm_get_sva_pasid() - get sva pasid for the vm if allocated
+ * @vm: The vm to get valid sva_handle
+ *
+ * Helper to get associated pasid for the vm which uses svm.
+ *
+ * Return: %pasid value on Success. -EINVAL if not pasid available.
+ */
+int xe_vm_get_sva_pasid(struct xe_vm *vm)
+{
+	if (!vm || !vm->sva_handle)
+		return -EINVAL;
+
+	return iommu_sva_get_pasid(vm->sva_handle);
+}
+
 static void xe_vm_sva_handle_init(struct xe_vm *vm)
 {
 	struct device *dev = vm->xe->drm.dev;
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index 89c4f95984a2..fe10fd91cdfa 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -290,6 +290,8 @@ struct dma_fence *xe_vm_bind_kernel_bo(struct xe_vm *vm, struct xe_bo *bo,
 
 void xe_vm_resume_rebind_worker(struct xe_vm *vm);
 
+int xe_vm_get_sva_pasid(struct xe_vm *vm);
+
 /**
  * xe_vm_resv() - Return's the vm's reservation object
  * @vm: The vm
-- 
2.43.0

[PATCH 4/6] drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts

[Nareshkumar Gollakoti]

Add CTX_SYS_PASID definitions to the LRC layout and program the field
during LRC context initialization when ATS and PASID are both enabled on
the PCI device.

When ATS/PASID support is available, fetch the VM SVA PASID and write it
to CTX_SYS_PASID together with the ATS enable and PASID valid bits. This
ensures the GPU context carries the system PASID state required for
address translation services.

Bspec: 67168

Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 ++++
 drivers/gpu/drm/xe/xe_lrc.c             | 37 +++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/gpu/drm/xe/regs/xe_lrc_layout.h b/drivers/gpu/drm/xe/regs/xe_lrc_layout.h
index 6f7abc0181b5..9394e3d5aecb 100644
--- a/drivers/gpu/drm/xe/regs/xe_lrc_layout.h
+++ b/drivers/gpu/drm/xe/regs/xe_lrc_layout.h
@@ -18,6 +18,7 @@
 #define CTX_TIMESTAMP_UDW		(0x24 + 1)
 #define CTX_INDIRECT_RING_STATE		(0x26 + 1)
 #define CTX_ACC_CTR_THOLD		(0x2a + 1)
+#define CTX_SYS_PASID			(0x2c + 1)
 #define CTX_ASID			(0x2e + 1)
 #define CTX_PDP0_UDW			(0x30 + 1)
 #define CTX_PDP0_LDW			(0x32 + 1)
@@ -45,4 +46,9 @@
 #define INDIRECT_CTX_RING_START_UDW	(0x08 + 1)
 #define INDIRECT_CTX_RING_CTL		(0x0a + 1)
 
+/* Fields for CTX_SYS_PASID */
+#define CTX_SYS_ATS_ENABLED                    REG_BIT(31)
+#define CTX_SYS_PASID_VALID                    REG_BIT(30)
+#define CTX_SYS_SYSTEM_PASID_MASK	       REG_GENMASK(19, 0)
+
 #endif
diff --git a/drivers/gpu/drm/xe/xe_lrc.c b/drivers/gpu/drm/xe/xe_lrc.c
index de9a9560ecf3..a4f7f0492262 100644
--- a/drivers/gpu/drm/xe/xe_lrc.c
+++ b/drivers/gpu/drm/xe/xe_lrc.c
@@ -9,6 +9,7 @@
 
 #include <linux/ascii85.h>
 #include <linux/panic.h>
+#include <linux/pci.h>
 
 #include "instructions/xe_mi_commands.h"
 #include "instructions/xe_gfxpipe_commands.h"
@@ -35,6 +36,7 @@
 #include "xe_trace_lrc.h"
 #include "xe_vm.h"
 #include "xe_wa.h"
+#include "xe_pci.h"
 
 #define LRC_VALID				BIT_ULL(0)
 #define LRC_PRIVILEGE				BIT_ULL(8)
@@ -998,6 +1000,16 @@ void xe_lrc_write_ctx_reg(struct xe_lrc *lrc, int reg_nr, u32 val)
 	xe_map_write32(xe, &map, val);
 }
 
+static void xe_lrc_rmw_ctx_reg(struct xe_lrc *lrc,
+			       int reg_nr, u32 mask, u32 val)
+{
+	u32 oldval = xe_lrc_read_ctx_reg(lrc, reg_nr);
+	u32 newval = (oldval & ~mask) | (val & mask);
+
+	if (newval != oldval)
+		xe_lrc_write_ctx_reg(lrc, reg_nr, newval);
+}
+
 static void *empty_lrc_data(struct xe_hw_engine *hwe)
 {
 	struct xe_gt *gt = hwe->gt;
@@ -1493,6 +1505,26 @@ static u8 xe_multi_queue_prio_to_lrc(struct xe_lrc *lrc, enum xe_multi_queue_pri
 	return priority;
 }
 
+static void xe_lrc_program_ctx_sys_pasid(struct xe_lrc *lrc, struct xe_vm *vm)
+{
+	int pasid;
+	u32 mask = CTX_SYS_ATS_ENABLED | CTX_SYS_PASID_VALID |
+		   CTX_SYS_SYSTEM_PASID_MASK;
+	struct xe_device *xe = lrc_to_xe(lrc);
+
+	pasid = xe_vm_get_sva_pasid(vm);
+	/* PASID 0 is reserved by the IOMMU for non-PASID-capable devices. */
+	if (pasid <= 0)
+		return;
+
+	xe_lrc_rmw_ctx_reg(lrc, CTX_SYS_PASID, mask,
+			   CTX_SYS_ATS_ENABLED |
+			   CTX_SYS_PASID_VALID |
+			   (pasid & CTX_SYS_SYSTEM_PASID_MASK));
+	drm_dbg(&xe->drm, "LRC: context enabled with ATS and PASID=%d for VM %p and lrc %p\n",
+		pasid, vm, lrc);
+}
+
 /**
  * xe_lrc_set_multi_queue_priority() - Set multi queue priority in LRC
  * @lrc: Logical Ring Context
@@ -1542,6 +1574,7 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct
 	struct xe_gt *gt = hwe->gt;
 	struct xe_tile *tile = gt_to_tile(gt);
 	struct xe_device *xe = gt_to_xe(gt);
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
 	struct iosys_map map;
 	u32 arb_enable;
 	u32 state_cache_perf_fix[3];
@@ -1642,6 +1675,10 @@ static int xe_lrc_ctx_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, struct
 	if (GRAPHICS_VER(xe) >= 20 && hwe->class == XE_ENGINE_CLASS_RENDER)
 		xe_lrc_set_gpgpu_preemption_level(lrc, gt);
 
+	/* Program CTX_SYS_PASID when ATS and PASID is supported. */
+	if (xe_pci_ats_pasid_enabled(pdev))
+		xe_lrc_program_ctx_sys_pasid(lrc, vm);
+
 	lrc->desc = LRC_VALID;
 	lrc->desc |= FIELD_PREP(LRC_ADDRESSING_MODE, LRC_LEGACY_64B_CONTEXT);
 	/* TODO: Priority */
-- 
2.43.0

[PATCH 5/6] drm/xe/pci: add helper for ATS/PASID without PRI

[Nareshkumar Gollakoti]

On platforms where ATS and PASID are enabled but PRI is not available,
SVA faults cannot rely on the IOMMU to fault in missing CPU mirror
bindings. Those faults must instead be recovered in driver.

Add a small PCI helper to report this condition so fault-handling code
can use a single predicate for the PRI-less SVA case.

Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/xe_pci.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_pci.h b/drivers/gpu/drm/xe/xe_pci.h
index 5f784229a5ec..cd22d316883b 100644
--- a/drivers/gpu/drm/xe/xe_pci.h
+++ b/drivers/gpu/drm/xe/xe_pci.h
@@ -18,4 +18,13 @@ static inline bool xe_pci_ats_pasid_enabled(struct pci_dev *pdev)
 	return pdev->ats_enabled && pdev->pasid_enabled;
 }
 
+/*
+ * True when SVA faults must be serviced in software: ATS/PASID is
+ * enabled but PRI is not available, so the IOMMU has no self-healing path.
+ */
+static inline bool xe_pci_svm_needs_sw_iopf(struct pci_dev *pdev)
+{
+	return xe_pci_ats_pasid_enabled(pdev) && !pdev->pri_enabled;
+}
+
 #endif
-- 
2.43.0

[PATCH 6/6] drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

[Nareshkumar Gollakoti]

On ATS/PASID platforms without PRI, xe cannot rely on the hardware
page request path to populate missing CPU mirror mappings for SVA faults.
This can leave xe_pagefault_service() without a VMA even when the faulting
address is valid in the process address space.

Create a CPU-address-mirror VMA on demand for such faults so they can
proceed through the existing SVM fault handling path.

Faults against unmapped addresses or unusable VMs continue to fail.

Signed-off-by: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>
---
 drivers/gpu/drm/xe/xe_pagefault.c       | 49 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
 2 files changed, 55 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_pagefault.c b/drivers/gpu/drm/xe/xe_pagefault.c
index 98cce0f12b12..c3acec28e115 100644
--- a/drivers/gpu/drm/xe/xe_pagefault.c
+++ b/drivers/gpu/drm/xe/xe_pagefault.c
@@ -17,6 +17,7 @@
 #include "xe_migrate.h"
 #include "xe_pagefault.h"
 #include "xe_pagefault_types.h"
+#include "xe_pci.h"
 #include "xe_pm.h"
 #include "xe_svm.h"
 #include "xe_trace_bo.h"
@@ -268,6 +269,7 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
 {
 	struct xe_gt *gt = pf->gt;
 	struct xe_device *xe = gt_to_xe(gt);
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
 	struct xe_vm *vm;
 	struct xe_vma *vma = NULL;
 	int err;
@@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
 	}
 
 	vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+
+	/*
+	 * ATS/PASID is negotiated but this hardware has no PRI, so there is
+	 * no IOMMU self-healing path to fault the page in - the driver has
+	 * to create the missing CPU-address-mirror binding itself and let
+	 * the SVM path below service the fault.
+	 */
+	if (!vma && vm->sva_handle && xe_pci_svm_needs_sw_iopf(pdev)) {
+		u64 page_size = vm->flags & XE_VM_FLAG_64K ? SZ_64K : SZ_4K;
+		u64 start = ALIGN_DOWN(xe_pagefault_addr(pf), page_size);
+		u64 end = start + page_size;
+		int alloc_err = 0;
+
+		up_read(&vm->lock);
+		down_write(&vm->lock);
+		/* Lock was dropped, re-check: a racing fault may have won */
+		vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+		if (!vma) {
+			if (xe_vm_is_closed_or_banned(vm)) {
+				alloc_err = -ENOENT;
+			} else {
+				/* Merge with adjacent mirror VMAs instead of leaving singletons */
+				xe_vm_find_cpu_addr_mirror_vma_range(vm, &start, &end);
+				alloc_err = xe_vm_alloc_cpu_addr_mirror_vma(vm, start, end - start);
+			}
+			if (!alloc_err)
+				vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
+		}
+		downgrade_write(&vm->lock);
+
+		if (alloc_err || !vma || !xe_vma_is_cpu_addr_mirror(vma)) {
+			/*
+			 * err is what actually tells guc_ack_fault() to NACK;
+			 * distinguish a transient allocation failure from a
+			 * genuinely unmapped/closed address for diagnostics.
+			 */
+			err = alloc_err ? : -EINVAL;
+			if (alloc_err == -ENOMEM)
+				xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_ALLOC);
+			else
+				xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_NOT_FOUND);
+			goto unlock_vm;
+		}
+	}
+
 	if (!vma) {
 		err = -EINVAL;
 		xe_pagefault_set_error(pf, XE_PAGEFAULT_ERROR_VMA_NOT_FOUND);
@@ -589,6 +636,8 @@ static const char *xe_pagefault_error_to_str(enum xe_pagefault_error error)
 		return "VM_CLOSED";
 	case XE_PAGEFAULT_ERROR_VMA_NOT_FOUND:
 		return "VMA_NOT_FOUND";
+	case XE_PAGEFAULT_ERROR_VMA_ALLOC:
+		return "VMA_ALLOC";
 	case XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION:
 		return "READ_ONLY_VIOLATION";
 	case XE_PAGEFAULT_ERROR_VMA_NEEDS_VRAM_CHECK:
diff --git a/drivers/gpu/drm/xe/xe_pagefault_types.h b/drivers/gpu/drm/xe/xe_pagefault_types.h
index 56defc05922e..d95d47ee0c0c 100644
--- a/drivers/gpu/drm/xe/xe_pagefault_types.h
+++ b/drivers/gpu/drm/xe/xe_pagefault_types.h
@@ -59,6 +59,12 @@ enum xe_pagefault_error {
 	XE_PAGEFAULT_ERROR_VM_CLOSED,
 	/** @XE_PAGEFAULT_ERROR_VMA_NOT_FOUND: No VMA covers the faulted address */
 	XE_PAGEFAULT_ERROR_VMA_NOT_FOUND,
+	/**
+	 * @XE_PAGEFAULT_ERROR_VMA_ALLOC: Failed to allocate a CPU-address-mirror
+	 * VMA to service a PRI-less SVA fault (e.g. -ENOMEM); distinct from a
+	 * genuinely unmapped address
+	 */
+	XE_PAGEFAULT_ERROR_VMA_ALLOC,
 	/**
 	 * @XE_PAGEFAULT_ERROR_READ_ONLY_VIOLATION: Write/atomic fault on a
 	 * read-only VMA
-- 
2.43.0

Re: [PATCH 6/6] drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] drm/xe: NULL deref and uninitialized PAT in xe_vm_alloc_cpu_addr_mirror_vma()
- [High] drm/xe: 4K VMA check before 64K mirror alloc in xe_pagefault_service()
--

commit c441331b0c386c437e9ad479c73465c57bd86054
Author: Nareshkumar Gollakoti <naresh.kumar.g@intel.com>

drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI

This commit adds on-demand creation of a cpu-address-mirror vma in
xe_pagefault_service() when servicing sva faults on platforms with ats/pasid
enabled but without pri support. It also introduces the
XE_PAGEFAULT_ERROR_VMA_ALLOC error code to report vma allocation failures.

> diff --git a/drivers/gpu/drm/xe/xe_pagefault.c b/drivers/gpu/drm/xe/xe_pagefault.c
> index 5247321c805ec..fbdf12d3d5580 100644
> --- a/drivers/gpu/drm/xe/xe_pagefault.c
> +++ b/drivers/gpu/drm/xe/xe_pagefault.c
[ ... ]
> @@ -298,6 +300,51 @@ static int xe_pagefault_service(struct xe_pagefault *pf)
>  	}
>  
>  	vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));
> +
> +	/*
> +	 * ATS/PASID is negotiated but this hardware has no PRI, so there is
> +	 * no IOMMU self-healing path to fault the page in - the driver has
> +	 * to create the missing CPU-address-mirror binding itself and let
> +	 * the SVM path below service the fault.
> +	 */
> +	if (!vma && vm->sva_handle && xe_pci_svm_needs_sw_iopf(pdev)) {
> +		u64 page_size = vm->flags & XE_VM_FLAG_64K ? SZ_64K : SZ_4K;
> +		u64 start = ALIGN_DOWN(xe_pagefault_addr(pf), page_size);
> +		u64 end = start + page_size;
> +		int alloc_err = 0;
> +
> +		up_read(&vm->lock);
> +		down_write(&vm->lock);
> +		/* Lock was dropped, re-check: a racing fault may have won */
> +		vma = xe_vm_find_vma_by_addr(vm, xe_pagefault_addr(pf));

[Severity: High]
When vm->flags & XE_VM_FLAG_64K is set, xe_pagefault_service() aligns start
and end to SZ_64K, but xe_vm_find_vma_by_addr() in drivers/gpu/drm/xe/xe_vm.c
only checks a 4k range at the faulting address:

drivers/gpu/drm/xe/xe_vm.c:xe_vm_find_vma_by_addr() {
	return xe_vm_find_overlapping_vma(vm, page_addr, SZ_4K);
}

Because vm_bind_ioctl_check_args() in drivers/gpu/drm/xe/xe_vm.c and
madvise_args_are_sane() in drivers/gpu/drm/xe/xe_vm_madvise.c allow
4k-aligned mappings on 64k vms (xe_vm_bind_ioctl_validate_bo() only requires
64k alignment for bos with XE_BO_FLAG_INTERNAL_64K), another 4k-aligned vma
(such as a system-memory bo, userptr, sparse vma, or madvise-split vma) can
already exist elsewhere within [start, end).

If a fault hits an unmapped 4k page in that 64k window without first checking
xe_vm_find_overlapping_vma(vm, start, page_size), won't
xe_vm_alloc_cpu_addr_mirror_vma() -> xe_vm_alloc_vma() ->
drm_gpuvm_sm_map_ops_create() generate unmap or remap operations that destroy
the overlapping vma via xe_vma_destroy() without unbinding its page tables or
holding bo->ttm.base.resv, triggering xe_bo_assert_held(bo) in
xe_vma_destroy()?

> +		if (!vma) {
> +			if (xe_vm_is_closed_or_banned(vm)) {
> +				alloc_err = -ENOENT;
> +			} else {
> +				/* Merge with adjacent mirror VMAs instead of leaving singletons */
> +				xe_vm_find_cpu_addr_mirror_vma_range(vm, &start, &end);
> +				alloc_err = xe_vm_alloc_cpu_addr_mirror_vma(vm, start, end - start);

[Severity: Critical]
What happens in xe_pagefault_service() when a fault occurs at an address with
no immediately adjacent default-attribute mirror vma at start - SZ_4K or
end + 1 (such as the first sva page fault in the vm)?

In that case, xe_vm_find_cpu_addr_mirror_vma_range() leaves [start, end)
unchanged so the range does not overlap any existing gpuva in vm->gpuvm.

When xe_vm_alloc_cpu_addr_mirror_vma() calls xe_vm_alloc_vma(vm, &map_req,
false) in drivers/gpu/drm/xe/xe_vm.c, vma_flags is initialized to 0 and
default_pat is left uninitialized on the stack, because both are only
populated from an existing vma during DRM_GPUVA_OP_UNMAP or
DRM_GPUVA_OP_REMAP:

drivers/gpu/drm/xe/xe_vm.c:xe_vm_alloc_vma() {
	unsigned int vma_flags = 0;
	bool remap_op = false;
	struct xe_vma_mem_attr tmp_attr = {};
	u16 default_pat;
	...
		if (!is_madvise) {
			if (__op->op == DRM_GPUVA_OP_UNMAP) {
				vma = gpuva_to_vma(op->base.unmap.va);
				XE_WARN_ON(!xe_vma_has_default_mem_attrs(vma));
				default_pat = vma->attr.default_pat_index;
				vma_flags = vma->gpuva.flags;
			}

			if (__op->op == DRM_GPUVA_OP_REMAP) {
				vma = gpuva_to_vma(op->base.remap.unmap->va);
				default_pat = vma->attr.default_pat_index;
				vma_flags = vma->gpuva.flags;
			}

			if (__op->op == DRM_GPUVA_OP_MAP) {
				op->map.vma_flags |= vma_flags & XE_VMA_CREATE_MASK;
				op->map.pat_index = default_pat;
			}
		}
}

Because drm_gpuvm_sm_map_ops_create() only emits DRM_GPUVA_OP_MAP when no
existing gpuva overlaps [start, end), op->map.vma_flags remains 0 (missing
XE_VMA_SYSTEM_ALLOCATOR) and op->map.pat_index uses the uninitialized
default_pat variable.

From there, vm_bind_ioctl_ops_parse() -> new_vma() -> xe_vma_create() in
drivers/gpu/drm/xe/xe_vm.c sees !bo && !is_null && !is_cpu_addr_mirror,
allocates a struct xe_userptr_vma instead of a mirror vma, and calls
xe_userptr_setup() in drivers/gpu/drm/xe/xe_userptr.c:

drivers/gpu/drm/xe/xe_userptr.c:xe_userptr_setup() {
	...
	err = mmu_interval_notifier_insert(&userptr->notifier, current->mm,
					   start, range,
					   &vma_userptr_notifier_ops);
}

Since xe_pagefault_service() runs on a workqueue kthread where current->mm is
NULL, won't this cause a null pointer dereference inside
mmu_interval_notifier_insert()?

> +			}
[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261021041117.1774877-1-naresh.kumar.g@intel.com?part=6

✗ CI.KUnit: failure for ATS/PASID SVA support and PRI-less fault servicing

[Patchwork]

== Series Details ==

Series: ATS/PASID SVA support and PRI-less fault servicing
URL   : https://patchwork.freedesktop.org/series/176519/
State : failure

== Summary ==

+ trap cleanup EXIT
+ kunitconfigs=('/kernel/drivers/gpu/tests/.kunitconfig' '/kernel/drivers/gpu/drm/xe/.kunitconfig' '/kernel/drivers/gpu/drm/tests/.kunitconfig' '/kernel/drivers/gpu/drm/ttm/tests/.kunitconfig' '/kernel/drivers/dma-buf/.kunitconfig')
+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/tests/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/tests/.kunitconfig
[04:20:43] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
[04:20:47] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
[04:21:08] Starting KUnit Kernel (1/1)...
[04:21:08] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[04:21:08] ================= gpu_buddy (14 subtests) ==================
[04:21:08] [PASSED] gpu_test_buddy_alloc_limit
[04:21:08] [PASSED] gpu_test_buddy_alloc_optimistic
[04:21:08] [PASSED] gpu_test_buddy_alloc_pessimistic
[04:21:08] [PASSED] gpu_test_buddy_alloc_pathological
[04:21:08] [PASSED] gpu_test_buddy_alloc_contiguous
[04:21:08] [PASSED] gpu_test_buddy_alloc_clear
[04:21:08] [PASSED] gpu_test_buddy_alloc_range
[04:21:09] [PASSED] gpu_test_buddy_alloc_range_bias
[04:21:10] [PASSED] gpu_test_buddy_fragmentation_performance
[04:21:10] [PASSED] gpu_test_buddy_dirty_tracker_performance
[04:21:10] [PASSED] gpu_test_buddy_alloc_exceeds_max_order
[04:21:10] [PASSED] gpu_test_buddy_offset_aligned_allocation
[04:21:10] [PASSED] gpu_test_buddy_subtree_offset_alignment_stress
[04:21:10] [PASSED] gpu_test_buddy_addr_to_block
[04:21:10] ==================== [PASSED] gpu_buddy ====================
[04:21:10] ============================================================
[04:21:10] Testing complete. Ran 14 tests: passed: 14
[04:21:10] Elapsed time: 27.450s total, 4.472s configuring, 20.860s building, 2.054s running

+ for kcfg in "${kunitconfigs[@]}"
+ [[ ! -f /kernel/drivers/gpu/drm/xe/.kunitconfig ]]
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig
ERROR:root:../drivers/gpu/drm/xe/xe_pci.c: In function ‘__xe_pci_probe’:
../drivers/gpu/drm/xe/xe_pci.c:1288:19: error: ‘struct pci_dev’ has no member named ‘pasid_cap’; did you mean ‘pcie_cap’?
 1288 |         if (pdev->pasid_cap)
      |                   ^~~~~~~~~
      |                   pcie_cap
make[7]: *** [../scripts/Makefile.build:290: drivers/gpu/drm/xe/xe_pci.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:551: drivers/gpu/drm/xe] Error 2
make[5]: *** [../scripts/Makefile.build:551: drivers/gpu/drm] Error 2
make[4]: *** [../scripts/Makefile.build:551: drivers/gpu] Error 2
make[3]: *** [../scripts/Makefile.build:551: drivers] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:2229: .] Error 2
make[1]: *** [/kernel/Makefile:248: __sub-make] Error 2
make: *** [Makefile:248: __sub-make] Error 2

[04:21:10] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
[04:21:12] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel

Re: [PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[Thomas Hellström]

Hi!

On Wed, 2026-10-21 at 09:41 +0530, Nareshkumar Gollakoti wrote:
> This series adds initial Shared Virtual Addressing (SVA) support for
> xe
> on platforms where ATS and PASID are available but PRI is not enabled
> or
> not supported.
> 
> For SVA, the device must be able to issue memory transactions tagged
> with a PASID so that address translation is performed in the process
> address space rather than in a driver owned address space. This
> series
> adds the xe side infrastructure needed for that flow: VM support for
> SVA
> PASID state, PASID retriveal helpers, and programming of
> CTX_SYS_PASID
> for ATS and PASID enabled contexts.
> 
> On systems with PRI, device page faults can be surfaced and resolved
> through the hardware page request interface, so the driver does not
> need
> to implement a separate fault servicing path for SVA.
> On non PRI systems, that hardware assisted fault path is
> unavailable.In
> that case, the driver still needs a way to resolve faults against the
> backing process address space so execution can proceed.
> 
> To support that case, this series introduces a CPU mirror VMA and
> uses
> it as part of a driver managed SVM path for SVA faults without
> PRI.The
> purpose of the CPU mirror VMA is to provide a CPU side VMA
> representation for the address range being accesses through SVA,so
> the
> driver can rely on the kernel MM/VMA machinery to validate mappings
> and
> resolve backing pages for fault handling in the absence of PRI.In
> other words, since the device cannot issue PRI requests, the driver
> needs a CPU visible VMA context that mirrors the GPU SVA range
> closely
> enough to let faults be service through normal MM paths.
> 
> At a high level, this series does the following:
> 
>  1. log ATS/PASID enablement state during probe,
>  2. add VM infrastructure needed for SVA PASID support,
>  3. add a helper to retrieve the VM SVA PASID,
>  4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
>  5. add a PCI helper for ATS/PASID operation without PRI,
>  6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
> 
> Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
> this flow and have been validated internally.Patch 6 adds the non PRI
> fault servicing path, and feedback on that overall approach would be
> especially appreciated.

I was under the impression that we'd integrate this with current SVM so
that we can still migrate to VRAM and use page-table mirroring or keep
data in system memory and use SVA page-table sharing *OR* page-table-
mirroring depending on user-space's wish?

Thanks,
Thomas


> 
> Himal Prasad Ghimiray (1):
>   drm/xe/vm: Setup SVA infrastructure for PASID support
> 
> Nareshkumar Gollakoti (5):
>   drm/xe/pci: log ATS/PASID enablement state at probe
>   drm/xe/vm: add helper to retrieve VM SVA PASID
>   drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
>   drm/xe/pci: add helper for ATS/PASID without PRI
>   drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
> 
>  drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
>  drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
>  drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
>  drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
>  drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
>  drivers/gpu/drm/xe/xe_vm.c              | 56
> +++++++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_vm.h              |  2 +
>  drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
>  9 files changed, 184 insertions(+)

Re: [PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[Kumar G, Naresh]

[-- Attachment #1: Type: text/plain, Size: 4830 bytes --]

Hi Thomas,

On 10/21/2026 8:55 PM, Thomas Hellström wrote:
> Hi!
>
> On Wed, 2026-10-21 at 09:41 +0530, Nareshkumar Gollakoti wrote:
>> This series adds initial Shared Virtual Addressing (SVA) support for
>> xe
>> on platforms where ATS and PASID are available but PRI is not enabled
>> or
>> not supported.
>>
>> For SVA, the device must be able to issue memory transactions tagged
>> with a PASID so that address translation is performed in the process
>> address space rather than in a driver owned address space. This
>> series
>> adds the xe side infrastructure needed for that flow: VM support for
>> SVA
>> PASID state, PASID retriveal helpers, and programming of
>> CTX_SYS_PASID
>> for ATS and PASID enabled contexts.
>>
>> On systems with PRI, device page faults can be surfaced and resolved
>> through the hardware page request interface, so the driver does not
>> need
>> to implement a separate fault servicing path for SVA.
>> On non PRI systems, that hardware assisted fault path is
>> unavailable.In
>> that case, the driver still needs a way to resolve faults against the
>> backing process address space so execution can proceed.
>>
>> To support that case, this series introduces a CPU mirror VMA and
>> uses
>> it as part of a driver managed SVM path for SVA faults without
>> PRI.The
>> purpose of the CPU mirror VMA is to provide a CPU side VMA
>> representation for the address range being accesses through SVA,so
>> the
>> driver can rely on the kernel MM/VMA machinery to validate mappings
>> and
>> resolve backing pages for fault handling in the absence of PRI.In
>> other words, since the device cannot issue PRI requests, the driver
>> needs a CPU visible VMA context that mirrors the GPU SVA range
>> closely
>> enough to let faults be service through normal MM paths.
>>
>> At a high level, this series does the following:
>>
>>   1. log ATS/PASID enablement state during probe,
>>   2. add VM infrastructure needed for SVA PASID support,
>>   3. add a helper to retrieve the VM SVA PASID,
>>   4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
>>   5. add a PCI helper for ATS/PASID operation without PRI,
>>   6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
>>
>> Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
>> this flow and have been validated internally.Patch 6 adds the non PRI
>> fault servicing path, and feedback on that overall approach would be
>> especially appreciated.
> I was under the impression that we'd integrate this with current SVM so
> that we can still migrate to VRAM and use page-table mirroring or keep
> data in system memory and use SVA page-table sharing *OR* page-table-
> mirroring depending on user-space's wish?
>
> Thanks,
> Thomas

IMO, Yes, that is the intended model. Userspace picks between the two 
approaches with the bind.

Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional VRAM 
migrate, PPGTT mirroring and this path is unchanged.

Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables 
with ATS+PASID. Backing has to stay in system memory. This is the 
end-to-end share flow, and it needs ATS+PASID+PRI.

If PRI is missing, an ATS miss cannot be healed by the IOMMU. The device 
then reports a GPU page fault. Instead of NACKing that unbound fault, we 
create a CPU-mirror VMA and service it with the same SVM path so the GPU 
gets a PPGTT mapping.


So, ATS+PASID+PRI → share works as user space asked (no bind). Bind 
with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the 
recovery path when share cannot complete at the IOMMU*.(This is one of 
the proposal)


Let me know if anything am missing.


Thanks,

Naresh

>
>> Himal Prasad Ghimiray (1):
>>    drm/xe/vm: Setup SVA infrastructure for PASID support
>>
>> Nareshkumar Gollakoti (5):
>>    drm/xe/pci: log ATS/PASID enablement state at probe
>>    drm/xe/vm: add helper to retrieve VM SVA PASID
>>    drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
>>    drm/xe/pci: add helper for ATS/PASID without PRI
>>    drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
>>
>>   drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
>>   drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
>>   drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
>>   drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
>>   drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
>>   drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
>>   drivers/gpu/drm/xe/xe_vm.c              | 56
>> +++++++++++++++++++++++++
>>   drivers/gpu/drm/xe/xe_vm.h              |  2 +
>>   drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
>>   9 files changed, 184 insertions(+)

[-- Attachment #2: Type: text/html, Size: 6735 bytes --]

Re: [PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[Thomas Hellström]

On Thu, 2026-10-22 at 12:00 +0530, Kumar G, Naresh wrote:
> Hi Thomas,
> 
> On 10/21/2026 8:55 PM, Thomas Hellström wrote:
> > Hi!
> > 
> > On Wed, 2026-10-21 at 09:41 +0530, Nareshkumar Gollakoti wrote:
> > > This series adds initial Shared Virtual Addressing (SVA) support
> > > for
> > > xe
> > > on platforms where ATS and PASID are available but PRI is not
> > > enabled
> > > or
> > > not supported.
> > > 
> > > For SVA, the device must be able to issue memory transactions
> > > tagged
> > > with a PASID so that address translation is performed in the
> > > process
> > > address space rather than in a driver owned address space. This
> > > series
> > > adds the xe side infrastructure needed for that flow: VM support
> > > for
> > > SVA
> > > PASID state, PASID retriveal helpers, and programming of
> > > CTX_SYS_PASID
> > > for ATS and PASID enabled contexts.
> > > 
> > > On systems with PRI, device page faults can be surfaced and
> > > resolved
> > > through the hardware page request interface, so the driver does
> > > not
> > > need
> > > to implement a separate fault servicing path for SVA.
> > > On non PRI systems, that hardware assisted fault path is
> > > unavailable.In
> > > that case, the driver still needs a way to resolve faults against
> > > the
> > > backing process address space so execution can proceed.
> > > 
> > > To support that case, this series introduces a CPU mirror VMA and
> > > uses
> > > it as part of a driver managed SVM path for SVA faults without
> > > PRI.The
> > > purpose of the CPU mirror VMA is to provide a CPU side VMA
> > > representation for the address range being accesses through
> > > SVA,so
> > > the
> > > driver can rely on the kernel MM/VMA machinery to validate
> > > mappings
> > > and
> > > resolve backing pages for fault handling in the absence of PRI.In
> > > other words, since the device cannot issue PRI requests, the
> > > driver
> > > needs a CPU visible VMA context that mirrors the GPU SVA range
> > > closely
> > > enough to let faults be service through normal MM paths.
> > > 
> > > At a high level, this series does the following:
> > > 
> > >   1. log ATS/PASID enablement state during probe,
> > >   2. add VM infrastructure needed for SVA PASID support,
> > >   3. add a helper to retrieve the VM SVA PASID,
> > >   4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
> > >   5. add a PCI helper for ATS/PASID operation without PRI,
> > >   6. add a CPU mirror VMA for handling SVA faults on non PRI
> > > platforms
> > > 
> > > Patches 1-5 provide the ATS/PASID and VM/context plumbing needed
> > > for
> > > this flow and have been validated internally.Patch 6 adds the non
> > > PRI
> > > fault servicing path, and feedback on that overall approach would
> > > be
> > > especially appreciated.
> > I was under the impression that we'd integrate this with current
> > SVM so
> > that we can still migrate to VRAM and use page-table mirroring or
> > keep
> > data in system memory and use SVA page-table sharing *OR* page-
> > table-
> > mirroring depending on user-space's wish?
> > 
> > Thanks,
> > Thomas
> 
> IMO, Yes, that is the intended model. Userspace picks between the two
> approaches with the bind.
> 
> Bind with CPU_ADDR_MIRROR → existing SVM. GPU page fault, optional
> VRAM 
> migrate, PPGTT mirroring and this path is unchanged.
> 
> Do not bind → SVA page-table sharing. IOMMU walks the CPU page tables
> with ATS+PASID. Backing has to stay in system memory. This is the 
> end-to-end share flow, and it needs ATS+PASID+PRI.
> 
> If PRI is missing, an ATS miss cannot be healed by the IOMMU. The
> device 
> then reports a GPU page fault. Instead of NACKing that unbound fault,
> we 
> create a CPU-mirror VMA and service it with the same SVM path so the
> GPU 
> gets a PPGTT mapping.
> 
> 
> So, ATS+PASID+PRI → share works as user space asked (no bind). Bind 
> with CPU_ADDR_MIRROR → SVM, independent of PRI. *PRI-less is only the
> recovery path when share cannot complete at the IOMMU*.(This is one
> of 
> the proposal)
> 
> 
> Let me know if anything am missing.

Yes, first on the (implicit) UAPI, KMD must never implicitly create a
GPU_VMA. This is done to be able to handle page-faults for non-PRI
capable hardware. Couldn't that be done much easier with a separate
page-fault handler that just calls fixup_user_fault() or a similar in-
kernel API and mark the gpu fault as handled? Then I assume the
ATS+PASID request will be retried?

On the implicit "Everything not having a GPU vma will be using
ATS/PASID" I strongly oppose that. Was this discussed on the UAPI
channel? Does NEO have any patches published to back this up?
ATS/PASID enables us to HW optimize a subset of the existing SVM
functionality, and IMO KMD is the best judge of whether to use it or
not, possibly with a UMD madvise hint. IMO we should require a
CPU_ADDR_MIRROR vma covering also the ATS/PASID regions and add an
MADVISE setting for user-space to prefer ATS/PASID lookups over current
hmm mirroring. If user-space then selects an MADVISE setting that is
incompatible with ATS/PASID (like different PAT, VRAM preferred region
etc.) Then KMD can transparently move over to that. We can also reuse
existing igts and user-space. If UMD needs to detect whether ATS/PASID
is usable on the VM then I figure we also must consider that.

Finally, it looks like you don't use the XE_PAGE_FORCE_FAULT bit, like
in here:

https://patchwork.freedesktop.org/patch/657641/?series=149886&rev=3

How would you then stop the hardware from attempting an ATS/PASID
lookup rather than faulting in mapped VMA?

Thanks,
Thomas


> 
> 
> Thanks,
> 
> Naresh
> 
> > 
> > > Himal Prasad Ghimiray (1):
> > >    drm/xe/vm: Setup SVA infrastructure for PASID support
> > > 
> > > Nareshkumar Gollakoti (5):
> > >    drm/xe/pci: log ATS/PASID enablement state at probe
> > >    drm/xe/vm: add helper to retrieve VM SVA PASID
> > >    drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled
> > > contexts
> > >    drm/xe/pci: add helper for ATS/PASID without PRI
> > >    drm/xe/pagefault: add CPU mirror VMA for SVA faults without
> > > PRI
> > > 
> > >   drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
> > >   drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
> > >   drivers/gpu/drm/xe/xe_pagefault.c       | 49
> > > ++++++++++++++++++++++
> > >   drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
> > >   drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
> > >   drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
> > >   drivers/gpu/drm/xe/xe_vm.c              | 56
> > > +++++++++++++++++++++++++
> > >   drivers/gpu/drm/xe/xe_vm.h              |  2 +
> > >   drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
> > >   9 files changed, 184 insertions(+)

Re: [PATCH 0/6] ATS/PASID SVA support and PRI-less fault servicing

[Matthew Brost]

On Wed, Oct 21, 2026 at 09:41:11AM +0530, Nareshkumar Gollakoti wrote:
> This series adds initial Shared Virtual Addressing (SVA) support for xe
> on platforms where ATS and PASID are available but PRI is not enabled or
> not supported.
> 
> For SVA, the device must be able to issue memory transactions tagged
> with a PASID so that address translation is performed in the process
> address space rather than in a driver owned address space. This series
> adds the xe side infrastructure needed for that flow: VM support for SVA
> PASID state, PASID retriveal helpers, and programming of CTX_SYS_PASID
> for ATS and PASID enabled contexts.
> 
> On systems with PRI, device page faults can be surfaced and resolved
> through the hardware page request interface, so the driver does not need
> to implement a separate fault servicing path for SVA.
> On non PRI systems, that hardware assisted fault path is unavailable.In
> that case, the driver still needs a way to resolve faults against the
> backing process address space so execution can proceed.
> 
> To support that case, this series introduces a CPU mirror VMA and uses
> it as part of a driver managed SVM path for SVA faults without PRI.The
> purpose of the CPU mirror VMA is to provide a CPU side VMA
> representation for the address range being accesses through SVA,so the
> driver can rely on the kernel MM/VMA machinery to validate mappings and
> resolve backing pages for fault handling in the absence of PRI.In
> other words, since the device cannot issue PRI requests, the driver
> needs a CPU visible VMA context that mirrors the GPU SVA range closely
> enough to let faults be service through normal MM paths.
> 
> At a high level, this series does the following:
> 
>  1. log ATS/PASID enablement state during probe,
>  2. add VM infrastructure needed for SVA PASID support,
>  3. add a helper to retrieve the VM SVA PASID,
>  4. program CTX_SYS_PASID for ATS/PASID-enabled contexts,
>  5. add a PCI helper for ATS/PASID operation without PRI,
>  6. add a CPU mirror VMA for handling SVA faults on non PRI platforms
> 
> Patches 1-5 provide the ATS/PASID and VM/context plumbing needed for
> this flow and have been validated internally.Patch 6 adds the non PRI
> fault servicing path, and feedback on that overall approach would be
> especially appreciated.
> 

Many concerns with this series, and I'm also generally quite confused
about the flows here. Let's take a step back and dig into what you're
trying to accomplish and why.

Prior to this series:

VM bind sets up PPGTTs.
- System memory uses DMA addresses (via dma-map) with DM=0 in the entry.
  These are GPAs (no pasid) translated by the IOMMU (or via ATS).
- Device memory uses DPAs with DM=1 in the entry.
SVM is configured via VM bind and CPU page-table mirroring.
- Pagefault or prefetches create mappings, data may or may not migrate.

After this series, all I have is questions and none are explained by the
cover-letter or patch commits:

1. Is a user expected to call VM bind? It seems the answer is yes.

2. Do we need any changes to the PPGTT setup? bspec53912 seems to
indicate that we do, but I'm not seeing any PPGTT setup changes in this
series. At the very minimum, I'd expect userptr to not use dma-address
and rely a PPGTT entry with P=0, FF=0 so the PASID walk occurs.

3. Are any programming model changes required in the UMD? If so, what
are they? I'm also concerned that this becomes a blind opt-in whenever
ATS + PASID are supported, as this would surely affect performance and a
UMD should have an explicit opt-in or opt-out mechanism.

4. Is SVM just implicitly enabled? Patch 6 seems to suggest that it is.
However, if I'm reading bspec53912 correctly and force-fault is not set
in the PPGTT, non-present PPGTT entries that correspond to valid CPU
pages would simply be translated and never migrated. If we miss in the
CPU page tables (e.g., the CPU has never touched the page), migration
could still occur, but only in 4K chunks, which would be terrible for
performance.

5. If SVM is explicitly enabled (e.g., through CPU page-table mirroring
via a CPU bind), we still have all the same issues described in #4 with
migration largely being bypassed. However, we'd at least hit the CPU
mirror and could perform larger migrations.

6. The most important question is: why is this being posted at all?
Unless we have hard data showing a performance benefit, I'd prefer not
to carry this code in the upstream tree. I could see some very narrow
use cases (e.g., userptr or SVM with a preferred system-memory location)
where avoiding a DMA map might provide a small performance win. However,
we'd likely need to rework things so that FF=1 is the default SVM
configuration. Alternatively, if we wanted access-counter-driven
migration rather than first-touch migration, this might make sense. 

Matt


> Himal Prasad Ghimiray (1):
>   drm/xe/vm: Setup SVA infrastructure for PASID support
> 
> Nareshkumar Gollakoti (5):
>   drm/xe/pci: log ATS/PASID enablement state at probe
>   drm/xe/vm: add helper to retrieve VM SVA PASID
>   drm/xe/lrc: program CTX_SYS_PASID for ATS/PASID-enabled contexts
>   drm/xe/pci: add helper for ATS/PASID without PRI
>   drm/xe/pagefault: add CPU mirror VMA for SVA faults without PRI
> 
>  drivers/gpu/drm/xe/regs/xe_lrc_layout.h |  6 +++
>  drivers/gpu/drm/xe/xe_lrc.c             | 37 ++++++++++++++++
>  drivers/gpu/drm/xe/xe_pagefault.c       | 49 ++++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_pagefault_types.h |  6 +++
>  drivers/gpu/drm/xe/xe_pci.c             |  7 ++++
>  drivers/gpu/drm/xe/xe_pci.h             | 15 +++++++
>  drivers/gpu/drm/xe/xe_vm.c              | 56 +++++++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_vm.h              |  2 +
>  drivers/gpu/drm/xe/xe_vm_types.h        |  6 +++
>  9 files changed, 184 insertions(+)
> 
> -- 
> 2.43.0
>