[PATCH 0/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[PATCH 0/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
This adds an option to AMDGPU_VA_OPs to disable redundant implicit sync
that happens on sparse unmap or replace operations.

This has seen a significant improvement in stutter in Forza Horizon 5
and Forza Horizon 4. (As games that had significant issues in sparse
binding related stutter).

This patchset also address a tangential issue that some changes were
not being flushed immediately after the ioctls, but were deferred to be
processed on the next CS submission, which also results in stalling.
A refactor of state machine is included to achieve this.

Compared to the previous series [1], this specifically targets the VM
operations and keep everything else intact, including implicit sync on
kernel-initiated moves.

I've been able to pass a full Vulkan CTS run on Navi 10 with this.

Userspace code for this is available at [2] and a branch for the kernel
code is available at [3].

[1]: https://lore.kernel.org/all/20230821062005.109771-1-ishitatsuyuki@gmail.com/
[2]: https://gitlab.freedesktop.org/ishitatsuyuki/mesa/-/commits/vm-explicit-sync
[3]: https://github.com/ishitatsuyuki/linux/tree/explicit-sync-drm-misc-next

Tatsuyuki Ishi (6):
  drm/amdgpu: Don't implicit sync PRT maps.
  drm/amdgpu: Separate eviction from VM status.
  drm/amdgpu: Flush VM updates for split bindings eagerly.
  drm/amdgpu: Remove redundant state change after validation.
  drm/amdgpu: Add flag to disable implicit sync for GEM operations.
  drm/amdgpu: Bump amdgpu driver version.

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |  32 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |   7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 185 ++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |  27 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     |   1 +
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |  18 +-
 include/uapi/drm/amdgpu_drm.h                 |   2 +
 11 files changed, 165 insertions(+), 120 deletions(-)

-- 
2.42.0

[PATCH 0/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
This adds an option to AMDGPU_VA_OPs to disable redundant implicit sync
that happens on sparse unmap or replace operations.

This has seen a significant improvement in stutter in Forza Horizon 5
and Forza Horizon 4. (As games that had significant issues in sparse
binding related stutter).

This patchset also address a tangential issue that some changes were
not being flushed immediately after the ioctls, but were deferred to be
processed on the next CS submission, which also results in stalling.
A refactor of state machine is included to achieve this.

Compared to the previous series [1], this specifically targets the VM
operations and keep everything else intact, including implicit sync on
kernel-initiated moves.

I've been able to pass a full Vulkan CTS run on Navi 10 with this.

Userspace code for this is available at [2] and a branch for the kernel
code is available at [3].

[1]: https://lore.kernel.org/all/20230821062005.109771-1-ishitatsuyuki@gmail.com/
[2]: https://gitlab.freedesktop.org/ishitatsuyuki/mesa/-/commits/vm-explicit-sync
[3]: https://github.com/ishitatsuyuki/linux/tree/explicit-sync-drm-misc-next

Tatsuyuki Ishi (6):
  drm/amdgpu: Don't implicit sync PRT maps.
  drm/amdgpu: Separate eviction from VM status.
  drm/amdgpu: Flush VM updates for split bindings eagerly.
  drm/amdgpu: Remove redundant state change after validation.
  drm/amdgpu: Add flag to disable implicit sync for GEM operations.
  drm/amdgpu: Bump amdgpu driver version.

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |  32 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |   7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 185 ++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |  27 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     |   1 +
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |  18 +-
 include/uapi/drm/amdgpu_drm.h                 |   2 +
 11 files changed, 165 insertions(+), 120 deletions(-)

-- 
2.42.0

[PATCH 1/6] drm/amdgpu: Don't implicit sync PRT maps.

[Tatsuyuki Ishi]

These are considered map operations rather than unmap, and there is no
point of doing implicit synchronization here.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index f5daadcec865..7b9762f1cddd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -902,7 +902,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & AMDGPU_PTE_VALID))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
-- 
2.42.0

[PATCH 1/6] drm/amdgpu: Don't implicit sync PRT maps.

[Tatsuyuki Ishi]

These are considered map operations rather than unmap, and there is no
point of doing implicit synchronization here.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index f5daadcec865..7b9762f1cddd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -902,7 +902,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & AMDGPU_PTE_VALID))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
-- 
2.42.0

[PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Tatsuyuki Ishi]

In short, eviction never really belonged to the vm_status state machine.
Even when evicted, the BO could belong to either the moved or done state.
The "evicted" state needed to handle both cases, causing greater confusion.

Additionally, there were inconsistencies in the definition of an evicted
BO. Some places are based on the `evict` parameter passed from the TTM move
callback, while the others were updated based on whether the BO got its
optimal placement. The second is more accurate for our use case. With this
refactor, the evicted state is solely determined by the second rule.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
 3 files changed, 29 insertions(+), 40 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 7b9762f1cddd..dd6f72e2a1d6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  * State for PDs/PTs and per VM BOs which are not at the location they should
  * be.
  */
-static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
+static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
 {
 	struct amdgpu_vm *vm = vm_bo->vm;
 	struct amdgpu_bo *bo = vm_bo->bo;
 
-	vm_bo->moved = true;
 	spin_lock(&vm_bo->vm->status_lock);
-	if (bo->tbo.type == ttm_bo_type_kernel)
-		list_move(&vm_bo->vm_status, &vm->evicted);
-	else
-		list_move_tail(&vm_bo->vm_status, &vm->evicted);
+	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
+		if (bo->tbo.type == ttm_bo_type_kernel)
+			list_move(&vm_bo->eviction_status, &vm->evicted);
+		else
+			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
+	} else {
+		list_del_init(&vm_bo->eviction_status);
+	}
 	spin_unlock(&vm_bo->vm->status_lock);
 }
+
 /**
  * amdgpu_vm_bo_moved - vm_bo is moved
  *
@@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 	base->bo = bo;
 	base->next = NULL;
 	INIT_LIST_HEAD(&base->vm_status);
+	INIT_LIST_HEAD(&base->eviction_status);
 
 	if (!bo)
 		return;
@@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 	 * is currently evicted. add the bo to the evicted list to make sure it
 	 * is validated on next vm use to avoid fault.
 	 * */
-	amdgpu_vm_bo_evicted(base);
+	amdgpu_vm_bo_set_evicted(base, true);
 }
 
 /**
@@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	while (!list_empty(&vm->evicted)) {
 		bo_base = list_first_entry(&vm->evicted,
 					   struct amdgpu_vm_bo_base,
-					   vm_status);
+					   eviction_status);
 		spin_unlock(&vm->status_lock);
 
 		bo = bo_base->bo;
@@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
 	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
-	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
+	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
@@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			return r;
 	}
 
-	/* If the BO is not in its preferred location add it back to
-	 * the evicted list so that it gets validated again on the
-	 * next command submission.
-	 */
-	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
-		uint32_t mem_type = bo->tbo.resource->mem_type;
-
-		if (!(bo->preferred_domains &
-		      amdgpu_mem_type_to_domain(mem_type)))
-			amdgpu_vm_bo_evicted(&bo_va->base);
-		else
-			amdgpu_vm_bo_idle(&bo_va->base);
-	} else {
+	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
+		amdgpu_vm_bo_idle(&bo_va->base);
+	else
 		amdgpu_vm_bo_done(&bo_va->base);
-	}
 
 	list_splice_init(&bo_va->invalids, &bo_va->valids);
 	bo_va->cleared = clear;
@@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 
 	spin_lock(&vm->status_lock);
 	list_del(&bo_va->base.vm_status);
+	list_del(&bo_va->base.eviction_status);
 	spin_unlock(&vm->status_lock);
 
 	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
@@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
 	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
 		bo = bo->parent;
 
+	/* If the BO is not in its preferred location add it back to
+	 * the evicted list so that it gets validated again on the
+	 * next command submission.
+	 */
+	uint32_t mem_type = bo->tbo.resource->mem_type;
+	bool suboptimal = !(bo->preferred_domains &
+			 amdgpu_mem_type_to_domain(mem_type));
+
 	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
 		struct amdgpu_vm *vm = bo_base->vm;
 
-		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
-			amdgpu_vm_bo_evicted(bo_base);
-			continue;
-		}
+		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
 
 		if (bo_base->moved)
 			continue;
@@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 {
 	struct amdgpu_bo_va *bo_va, *tmp;
 	u64 total_idle = 0;
-	u64 total_evicted = 0;
 	u64 total_relocated = 0;
 	u64 total_moved = 0;
 	u64 total_invalidated = 0;
 	u64 total_done = 0;
 	unsigned int total_idle_objs = 0;
-	unsigned int total_evicted_objs = 0;
 	unsigned int total_relocated_objs = 0;
 	unsigned int total_moved_objs = 0;
 	unsigned int total_invalidated_objs = 0;
@@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 	total_idle_objs = id;
 	id = 0;
 
-	seq_puts(m, "\tEvicted BOs:\n");
-	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
-		if (!bo_va->base.bo)
-			continue;
-		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
-	}
-	total_evicted_objs = id;
-	id = 0;
-
 	seq_puts(m, "\tRelocated BOs:\n");
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
 		if (!bo_va->base.bo)
@@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 
 	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
 		   total_idle_objs);
-	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
-		   total_evicted_objs);
 	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
 		   total_relocated_objs);
 	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 204ab13184ed..d9ab97eabda9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
 
 	/* protected by spinlock */
 	struct list_head		vm_status;
+	struct list_head		eviction_status;
 
 	/* protected by the BO being reserved */
 	bool				moved;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
index 96d601e209b8..f78f4040f466 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
 
 	spin_lock(&entry->vm->status_lock);
 	list_del(&entry->vm_status);
+	list_del(&entry->eviction_status);
 	spin_unlock(&entry->vm->status_lock);
 	amdgpu_bo_unref(&entry->bo);
 }
-- 
2.42.0

[PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Tatsuyuki Ishi]

In short, eviction never really belonged to the vm_status state machine.
Even when evicted, the BO could belong to either the moved or done state.
The "evicted" state needed to handle both cases, causing greater confusion.

Additionally, there were inconsistencies in the definition of an evicted
BO. Some places are based on the `evict` parameter passed from the TTM move
callback, while the others were updated based on whether the BO got its
optimal placement. The second is more accurate for our use case. With this
refactor, the evicted state is solely determined by the second rule.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
 3 files changed, 29 insertions(+), 40 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 7b9762f1cddd..dd6f72e2a1d6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  * State for PDs/PTs and per VM BOs which are not at the location they should
  * be.
  */
-static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
+static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
 {
 	struct amdgpu_vm *vm = vm_bo->vm;
 	struct amdgpu_bo *bo = vm_bo->bo;
 
-	vm_bo->moved = true;
 	spin_lock(&vm_bo->vm->status_lock);
-	if (bo->tbo.type == ttm_bo_type_kernel)
-		list_move(&vm_bo->vm_status, &vm->evicted);
-	else
-		list_move_tail(&vm_bo->vm_status, &vm->evicted);
+	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
+		if (bo->tbo.type == ttm_bo_type_kernel)
+			list_move(&vm_bo->eviction_status, &vm->evicted);
+		else
+			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
+	} else {
+		list_del_init(&vm_bo->eviction_status);
+	}
 	spin_unlock(&vm_bo->vm->status_lock);
 }
+
 /**
  * amdgpu_vm_bo_moved - vm_bo is moved
  *
@@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 	base->bo = bo;
 	base->next = NULL;
 	INIT_LIST_HEAD(&base->vm_status);
+	INIT_LIST_HEAD(&base->eviction_status);
 
 	if (!bo)
 		return;
@@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 	 * is currently evicted. add the bo to the evicted list to make sure it
 	 * is validated on next vm use to avoid fault.
 	 * */
-	amdgpu_vm_bo_evicted(base);
+	amdgpu_vm_bo_set_evicted(base, true);
 }
 
 /**
@@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	while (!list_empty(&vm->evicted)) {
 		bo_base = list_first_entry(&vm->evicted,
 					   struct amdgpu_vm_bo_base,
-					   vm_status);
+					   eviction_status);
 		spin_unlock(&vm->status_lock);
 
 		bo = bo_base->bo;
@@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
 	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
-	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
+	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
@@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			return r;
 	}
 
-	/* If the BO is not in its preferred location add it back to
-	 * the evicted list so that it gets validated again on the
-	 * next command submission.
-	 */
-	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
-		uint32_t mem_type = bo->tbo.resource->mem_type;
-
-		if (!(bo->preferred_domains &
-		      amdgpu_mem_type_to_domain(mem_type)))
-			amdgpu_vm_bo_evicted(&bo_va->base);
-		else
-			amdgpu_vm_bo_idle(&bo_va->base);
-	} else {
+	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
+		amdgpu_vm_bo_idle(&bo_va->base);
+	else
 		amdgpu_vm_bo_done(&bo_va->base);
-	}
 
 	list_splice_init(&bo_va->invalids, &bo_va->valids);
 	bo_va->cleared = clear;
@@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 
 	spin_lock(&vm->status_lock);
 	list_del(&bo_va->base.vm_status);
+	list_del(&bo_va->base.eviction_status);
 	spin_unlock(&vm->status_lock);
 
 	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
@@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
 	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
 		bo = bo->parent;
 
+	/* If the BO is not in its preferred location add it back to
+	 * the evicted list so that it gets validated again on the
+	 * next command submission.
+	 */
+	uint32_t mem_type = bo->tbo.resource->mem_type;
+	bool suboptimal = !(bo->preferred_domains &
+			 amdgpu_mem_type_to_domain(mem_type));
+
 	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
 		struct amdgpu_vm *vm = bo_base->vm;
 
-		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
-			amdgpu_vm_bo_evicted(bo_base);
-			continue;
-		}
+		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
 
 		if (bo_base->moved)
 			continue;
@@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 {
 	struct amdgpu_bo_va *bo_va, *tmp;
 	u64 total_idle = 0;
-	u64 total_evicted = 0;
 	u64 total_relocated = 0;
 	u64 total_moved = 0;
 	u64 total_invalidated = 0;
 	u64 total_done = 0;
 	unsigned int total_idle_objs = 0;
-	unsigned int total_evicted_objs = 0;
 	unsigned int total_relocated_objs = 0;
 	unsigned int total_moved_objs = 0;
 	unsigned int total_invalidated_objs = 0;
@@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 	total_idle_objs = id;
 	id = 0;
 
-	seq_puts(m, "\tEvicted BOs:\n");
-	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
-		if (!bo_va->base.bo)
-			continue;
-		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
-	}
-	total_evicted_objs = id;
-	id = 0;
-
 	seq_puts(m, "\tRelocated BOs:\n");
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
 		if (!bo_va->base.bo)
@@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 
 	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
 		   total_idle_objs);
-	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
-		   total_evicted_objs);
 	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
 		   total_relocated_objs);
 	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 204ab13184ed..d9ab97eabda9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
 
 	/* protected by spinlock */
 	struct list_head		vm_status;
+	struct list_head		eviction_status;
 
 	/* protected by the BO being reserved */
 	bool				moved;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
index 96d601e209b8..f78f4040f466 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
 
 	spin_lock(&entry->vm->status_lock);
 	list_del(&entry->vm_status);
+	list_del(&entry->eviction_status);
 	spin_unlock(&entry->vm->status_lock);
 	amdgpu_bo_unref(&entry->bo);
 }
-- 
2.42.0

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> In short, eviction never really belonged to the vm_status state machine.

I strongly disagree to that.

> Even when evicted, the BO could belong to either the moved or done state.
> The "evicted" state needed to handle both cases, causing greater confusion.
>
> Additionally, there were inconsistencies in the definition of an evicted
> BO. Some places are based on the `evict` parameter passed from the TTM move
> callback, while the others were updated based on whether the BO got its
> optimal placement. The second is more accurate for our use case. With this
> refactor, the evicted state is solely determined by the second rule.

That strongly sounds like you don't understand what the evicted state it 
good for.

The evicted state is for page directories, page tables and per VM BOs 
which needs to move around before doing the next CS.

Please further explain what you try to do here.

Regards,
Christian.

>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>   3 files changed, 29 insertions(+), 40 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 7b9762f1cddd..dd6f72e2a1d6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>    * State for PDs/PTs and per VM BOs which are not at the location they should
>    * be.
>    */
> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>   {
>   	struct amdgpu_vm *vm = vm_bo->vm;
>   	struct amdgpu_bo *bo = vm_bo->bo;
>   
> -	vm_bo->moved = true;
>   	spin_lock(&vm_bo->vm->status_lock);
> -	if (bo->tbo.type == ttm_bo_type_kernel)
> -		list_move(&vm_bo->vm_status, &vm->evicted);
> -	else
> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> +		if (bo->tbo.type == ttm_bo_type_kernel)
> +			list_move(&vm_bo->eviction_status, &vm->evicted);
> +		else
> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
> +	} else {
> +		list_del_init(&vm_bo->eviction_status);
> +	}
>   	spin_unlock(&vm_bo->vm->status_lock);
>   }
> +
>   /**
>    * amdgpu_vm_bo_moved - vm_bo is moved
>    *
> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   	base->bo = bo;
>   	base->next = NULL;
>   	INIT_LIST_HEAD(&base->vm_status);
> +	INIT_LIST_HEAD(&base->eviction_status);
>   
>   	if (!bo)
>   		return;
> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   	 * is currently evicted. add the bo to the evicted list to make sure it
>   	 * is validated on next vm use to avoid fault.
>   	 * */
> -	amdgpu_vm_bo_evicted(base);
> +	amdgpu_vm_bo_set_evicted(base, true);
>   }
>   
>   /**
> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   	while (!list_empty(&vm->evicted)) {
>   		bo_base = list_first_entry(&vm->evicted,
>   					   struct amdgpu_vm_bo_base,
> -					   vm_status);
> +					   eviction_status);
>   		spin_unlock(&vm->status_lock);
>   
>   		bo = bo_base->bo;
> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>   	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   			return r;
>   	}
>   
> -	/* If the BO is not in its preferred location add it back to
> -	 * the evicted list so that it gets validated again on the
> -	 * next command submission.
> -	 */
> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> -		uint32_t mem_type = bo->tbo.resource->mem_type;
> -
> -		if (!(bo->preferred_domains &
> -		      amdgpu_mem_type_to_domain(mem_type)))
> -			amdgpu_vm_bo_evicted(&bo_va->base);
> -		else
> -			amdgpu_vm_bo_idle(&bo_va->base);
> -	} else {
> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
> +		amdgpu_vm_bo_idle(&bo_va->base);
> +	else
>   		amdgpu_vm_bo_done(&bo_va->base);
> -	}
>   
>   	list_splice_init(&bo_va->invalids, &bo_va->valids);
>   	bo_va->cleared = clear;
> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>   
>   	spin_lock(&vm->status_lock);
>   	list_del(&bo_va->base.vm_status);
> +	list_del(&bo_va->base.eviction_status);
>   	spin_unlock(&vm->status_lock);
>   
>   	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>   	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>   		bo = bo->parent;
>   
> +	/* If the BO is not in its preferred location add it back to
> +	 * the evicted list so that it gets validated again on the
> +	 * next command submission.
> +	 */
> +	uint32_t mem_type = bo->tbo.resource->mem_type;
> +	bool suboptimal = !(bo->preferred_domains &
> +			 amdgpu_mem_type_to_domain(mem_type));
> +
>   	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>   		struct amdgpu_vm *vm = bo_base->vm;
>   
> -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> -			amdgpu_vm_bo_evicted(bo_base);
> -			continue;
> -		}
> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>   
>   		if (bo_base->moved)
>   			continue;
> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   {
>   	struct amdgpu_bo_va *bo_va, *tmp;
>   	u64 total_idle = 0;
> -	u64 total_evicted = 0;
>   	u64 total_relocated = 0;
>   	u64 total_moved = 0;
>   	u64 total_invalidated = 0;
>   	u64 total_done = 0;
>   	unsigned int total_idle_objs = 0;
> -	unsigned int total_evicted_objs = 0;
>   	unsigned int total_relocated_objs = 0;
>   	unsigned int total_moved_objs = 0;
>   	unsigned int total_invalidated_objs = 0;
> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   	total_idle_objs = id;
>   	id = 0;
>   
> -	seq_puts(m, "\tEvicted BOs:\n");
> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
> -		if (!bo_va->base.bo)
> -			continue;
> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
> -	}
> -	total_evicted_objs = id;
> -	id = 0;
> -
>   	seq_puts(m, "\tRelocated BOs:\n");
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>   		if (!bo_va->base.bo)
> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   
>   	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>   		   total_idle_objs);
> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
> -		   total_evicted_objs);
>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>   		   total_relocated_objs);
>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 204ab13184ed..d9ab97eabda9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>   
>   	/* protected by spinlock */
>   	struct list_head		vm_status;
> +	struct list_head		eviction_status;
>   
>   	/* protected by the BO being reserved */
>   	bool				moved;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> index 96d601e209b8..f78f4040f466 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>   
>   	spin_lock(&entry->vm->status_lock);
>   	list_del(&entry->vm_status);
> +	list_del(&entry->eviction_status);
>   	spin_unlock(&entry->vm->status_lock);
>   	amdgpu_bo_unref(&entry->bo);
>   }

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> In short, eviction never really belonged to the vm_status state machine.

I strongly disagree to that.

> Even when evicted, the BO could belong to either the moved or done state.
> The "evicted" state needed to handle both cases, causing greater confusion.
>
> Additionally, there were inconsistencies in the definition of an evicted
> BO. Some places are based on the `evict` parameter passed from the TTM move
> callback, while the others were updated based on whether the BO got its
> optimal placement. The second is more accurate for our use case. With this
> refactor, the evicted state is solely determined by the second rule.

That strongly sounds like you don't understand what the evicted state it 
good for.

The evicted state is for page directories, page tables and per VM BOs 
which needs to move around before doing the next CS.

Please further explain what you try to do here.

Regards,
Christian.

>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>   3 files changed, 29 insertions(+), 40 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 7b9762f1cddd..dd6f72e2a1d6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>    * State for PDs/PTs and per VM BOs which are not at the location they should
>    * be.
>    */
> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>   {
>   	struct amdgpu_vm *vm = vm_bo->vm;
>   	struct amdgpu_bo *bo = vm_bo->bo;
>   
> -	vm_bo->moved = true;
>   	spin_lock(&vm_bo->vm->status_lock);
> -	if (bo->tbo.type == ttm_bo_type_kernel)
> -		list_move(&vm_bo->vm_status, &vm->evicted);
> -	else
> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> +		if (bo->tbo.type == ttm_bo_type_kernel)
> +			list_move(&vm_bo->eviction_status, &vm->evicted);
> +		else
> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
> +	} else {
> +		list_del_init(&vm_bo->eviction_status);
> +	}
>   	spin_unlock(&vm_bo->vm->status_lock);
>   }
> +
>   /**
>    * amdgpu_vm_bo_moved - vm_bo is moved
>    *
> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   	base->bo = bo;
>   	base->next = NULL;
>   	INIT_LIST_HEAD(&base->vm_status);
> +	INIT_LIST_HEAD(&base->eviction_status);
>   
>   	if (!bo)
>   		return;
> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   	 * is currently evicted. add the bo to the evicted list to make sure it
>   	 * is validated on next vm use to avoid fault.
>   	 * */
> -	amdgpu_vm_bo_evicted(base);
> +	amdgpu_vm_bo_set_evicted(base, true);
>   }
>   
>   /**
> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   	while (!list_empty(&vm->evicted)) {
>   		bo_base = list_first_entry(&vm->evicted,
>   					   struct amdgpu_vm_bo_base,
> -					   vm_status);
> +					   eviction_status);
>   		spin_unlock(&vm->status_lock);
>   
>   		bo = bo_base->bo;
> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>   	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   			return r;
>   	}
>   
> -	/* If the BO is not in its preferred location add it back to
> -	 * the evicted list so that it gets validated again on the
> -	 * next command submission.
> -	 */
> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> -		uint32_t mem_type = bo->tbo.resource->mem_type;
> -
> -		if (!(bo->preferred_domains &
> -		      amdgpu_mem_type_to_domain(mem_type)))
> -			amdgpu_vm_bo_evicted(&bo_va->base);
> -		else
> -			amdgpu_vm_bo_idle(&bo_va->base);
> -	} else {
> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
> +		amdgpu_vm_bo_idle(&bo_va->base);
> +	else
>   		amdgpu_vm_bo_done(&bo_va->base);
> -	}
>   
>   	list_splice_init(&bo_va->invalids, &bo_va->valids);
>   	bo_va->cleared = clear;
> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>   
>   	spin_lock(&vm->status_lock);
>   	list_del(&bo_va->base.vm_status);
> +	list_del(&bo_va->base.eviction_status);
>   	spin_unlock(&vm->status_lock);
>   
>   	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>   	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>   		bo = bo->parent;
>   
> +	/* If the BO is not in its preferred location add it back to
> +	 * the evicted list so that it gets validated again on the
> +	 * next command submission.
> +	 */
> +	uint32_t mem_type = bo->tbo.resource->mem_type;
> +	bool suboptimal = !(bo->preferred_domains &
> +			 amdgpu_mem_type_to_domain(mem_type));
> +
>   	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>   		struct amdgpu_vm *vm = bo_base->vm;
>   
> -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
> -			amdgpu_vm_bo_evicted(bo_base);
> -			continue;
> -		}
> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>   
>   		if (bo_base->moved)
>   			continue;
> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   {
>   	struct amdgpu_bo_va *bo_va, *tmp;
>   	u64 total_idle = 0;
> -	u64 total_evicted = 0;
>   	u64 total_relocated = 0;
>   	u64 total_moved = 0;
>   	u64 total_invalidated = 0;
>   	u64 total_done = 0;
>   	unsigned int total_idle_objs = 0;
> -	unsigned int total_evicted_objs = 0;
>   	unsigned int total_relocated_objs = 0;
>   	unsigned int total_moved_objs = 0;
>   	unsigned int total_invalidated_objs = 0;
> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   	total_idle_objs = id;
>   	id = 0;
>   
> -	seq_puts(m, "\tEvicted BOs:\n");
> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
> -		if (!bo_va->base.bo)
> -			continue;
> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
> -	}
> -	total_evicted_objs = id;
> -	id = 0;
> -
>   	seq_puts(m, "\tRelocated BOs:\n");
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>   		if (!bo_va->base.bo)
> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   
>   	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>   		   total_idle_objs);
> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
> -		   total_evicted_objs);
>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>   		   total_relocated_objs);
>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 204ab13184ed..d9ab97eabda9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>   
>   	/* protected by spinlock */
>   	struct list_head		vm_status;
> +	struct list_head		eviction_status;
>   
>   	/* protected by the BO being reserved */
>   	bool				moved;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> index 96d601e209b8..f78f4040f466 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>   
>   	spin_lock(&entry->vm->status_lock);
>   	list_del(&entry->vm_status);
> +	list_del(&entry->eviction_status);
>   	spin_unlock(&entry->vm->status_lock);
>   	amdgpu_bo_unref(&entry->bo);
>   }

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Tatsuyuki Ishi]

> On Oct 31, 2023, at 22:55, Christian König <christian.koenig@amd.com> wrote:
> 
> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> In short, eviction never really belonged to the vm_status state machine.
> 
> I strongly disagree to that.
> 
>> Even when evicted, the BO could belong to either the moved or done state.
>> The "evicted" state needed to handle both cases, causing greater confusion.
>> 
>> Additionally, there were inconsistencies in the definition of an evicted
>> BO. Some places are based on the `evict` parameter passed from the TTM move
>> callback, while the others were updated based on whether the BO got its
>> optimal placement. The second is more accurate for our use case. With this
>> refactor, the evicted state is solely determined by the second rule.
> 
> That strongly sounds like you don't understand what the evicted state it good for.
> 
> The evicted state is for page directories, page tables and per VM BOs which needs to move around before doing the next CS.
> 
> Please further explain what you try to do here.

This is mainly an attempt to address inconsistency in the definition of “eviction”. The TTM move callback sets eviction when eviction happens through ttm_bo_evict. This is however not the only way a BO might end up outside its preferred domains.

amdgpu_vm_bo_update later updates the eviction state based on whether the BO is in its preferred domains. In my understanding this includes all cases where the BO is evicted through ttm_bo_evict. Therefore we should apply this definition right from the move callback, not only after amdgpu_vm_bo_update has been called at least once.

Tatsuyuki.

> 
> Regards,
> Christian.
> 
>> 
>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> ---
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>>  3 files changed, 29 insertions(+), 40 deletions(-)
>> 
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> index 7b9762f1cddd..dd6f72e2a1d6 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>   * State for PDs/PTs and per VM BOs which are not at the location they should
>>   * be.
>>   */
>> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
>> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>>  {
>>  	struct amdgpu_vm *vm = vm_bo->vm;
>>  	struct amdgpu_bo *bo = vm_bo->bo;
>>  -	vm_bo->moved = true;
>>  	spin_lock(&vm_bo->vm->status_lock);
>> -	if (bo->tbo.type == ttm_bo_type_kernel)
>> -		list_move(&vm_bo->vm_status, &vm->evicted);
>> -	else
>> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
>> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> +		if (bo->tbo.type == ttm_bo_type_kernel)
>> +			list_move(&vm_bo->eviction_status, &vm->evicted);
>> +		else
>> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
>> +	} else {
>> +		list_del_init(&vm_bo->eviction_status);
>> +	}
>>  	spin_unlock(&vm_bo->vm->status_lock);
>>  }
>> +
>>  /**
>>   * amdgpu_vm_bo_moved - vm_bo is moved
>>   *
>> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  	base->bo = bo;
>>  	base->next = NULL;
>>  	INIT_LIST_HEAD(&base->vm_status);
>> +	INIT_LIST_HEAD(&base->eviction_status);
>>    	if (!bo)
>>  		return;
>> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  	 * is currently evicted. add the bo to the evicted list to make sure it
>>  	 * is validated on next vm use to avoid fault.
>>  	 * */
>> -	amdgpu_vm_bo_evicted(base);
>> +	amdgpu_vm_bo_set_evicted(base, true);
>>  }
>>    /**
>> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  	while (!list_empty(&vm->evicted)) {
>>  		bo_base = list_first_entry(&vm->evicted,
>>  					   struct amdgpu_vm_bo_base,
>> -					   vm_status);
>> +					   eviction_status);
>>  		spin_unlock(&vm->status_lock);
>>    		bo = bo_base->bo;
>> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>  	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>>  		amdgpu_vm_bo_get_memory(bo_va, stats);
>>  -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
>> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>  		amdgpu_vm_bo_get_memory(bo_va, stats);
>>    	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  			return r;
>>  	}
>>  -	/* If the BO is not in its preferred location add it back to
>> -	 * the evicted list so that it gets validated again on the
>> -	 * next command submission.
>> -	 */
>> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> -		uint32_t mem_type = bo->tbo.resource->mem_type;
>> -
>> -		if (!(bo->preferred_domains &
>> -		      amdgpu_mem_type_to_domain(mem_type)))
>> -			amdgpu_vm_bo_evicted(&bo_va->base);
>> -		else
>> -			amdgpu_vm_bo_idle(&bo_va->base);
>> -	} else {
>> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
>> +		amdgpu_vm_bo_idle(&bo_va->base);
>> +	else
>>  		amdgpu_vm_bo_done(&bo_va->base);
>> -	}
>>    	list_splice_init(&bo_va->invalids, &bo_va->valids);
>>  	bo_va->cleared = clear;
>> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>    	spin_lock(&vm->status_lock);
>>  	list_del(&bo_va->base.vm_status);
>> +	list_del(&bo_va->base.eviction_status);
>>  	spin_unlock(&vm->status_lock);
>>    	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
>> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>  	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>>  		bo = bo->parent;
>>  +	/* If the BO is not in its preferred location add it back to
>> +	 * the evicted list so that it gets validated again on the
>> +	 * next command submission.
>> +	 */
>> +	uint32_t mem_type = bo->tbo.resource->mem_type;
>> +	bool suboptimal = !(bo->preferred_domains &
>> +			 amdgpu_mem_type_to_domain(mem_type));
>> +
>>  	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>>  		struct amdgpu_vm *vm = bo_base->vm;
>>  -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> -			amdgpu_vm_bo_evicted(bo_base);
>> -			continue;
>> -		}
>> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>>    		if (bo_base->moved)
>>  			continue;
>> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>  {
>>  	struct amdgpu_bo_va *bo_va, *tmp;
>>  	u64 total_idle = 0;
>> -	u64 total_evicted = 0;
>>  	u64 total_relocated = 0;
>>  	u64 total_moved = 0;
>>  	u64 total_invalidated = 0;
>>  	u64 total_done = 0;
>>  	unsigned int total_idle_objs = 0;
>> -	unsigned int total_evicted_objs = 0;
>>  	unsigned int total_relocated_objs = 0;
>>  	unsigned int total_moved_objs = 0;
>>  	unsigned int total_invalidated_objs = 0;
>> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>  	total_idle_objs = id;
>>  	id = 0;
>>  -	seq_puts(m, "\tEvicted BOs:\n");
>> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
>> -		if (!bo_va->base.bo)
>> -			continue;
>> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>> -	}
>> -	total_evicted_objs = id;
>> -	id = 0;
>> -
>>  	seq_puts(m, "\tRelocated BOs:\n");
>>  	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>  		if (!bo_va->base.bo)
>> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>    	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>  		   total_idle_objs);
>> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
>> -		   total_evicted_objs);
>>  	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>  		   total_relocated_objs);
>>  	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 204ab13184ed..d9ab97eabda9 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>>    	/* protected by spinlock */
>>  	struct list_head		vm_status;
>> +	struct list_head		eviction_status;
>>    	/* protected by the BO being reserved */
>>  	bool				moved;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> index 96d601e209b8..f78f4040f466 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>>    	spin_lock(&entry->vm->status_lock);
>>  	list_del(&entry->vm_status);
>> +	list_del(&entry->eviction_status);
>>  	spin_unlock(&entry->vm->status_lock);
>>  	amdgpu_bo_unref(&entry->bo);
>>  }
> 

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Tatsuyuki Ishi]

> On Oct 31, 2023, at 22:55, Christian König <christian.koenig@amd.com> wrote:
> 
> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> In short, eviction never really belonged to the vm_status state machine.
> 
> I strongly disagree to that.
> 
>> Even when evicted, the BO could belong to either the moved or done state.
>> The "evicted" state needed to handle both cases, causing greater confusion.
>> 
>> Additionally, there were inconsistencies in the definition of an evicted
>> BO. Some places are based on the `evict` parameter passed from the TTM move
>> callback, while the others were updated based on whether the BO got its
>> optimal placement. The second is more accurate for our use case. With this
>> refactor, the evicted state is solely determined by the second rule.
> 
> That strongly sounds like you don't understand what the evicted state it good for.
> 
> The evicted state is for page directories, page tables and per VM BOs which needs to move around before doing the next CS.
> 
> Please further explain what you try to do here.

This is mainly an attempt to address inconsistency in the definition of “eviction”. The TTM move callback sets eviction when eviction happens through ttm_bo_evict. This is however not the only way a BO might end up outside its preferred domains.

amdgpu_vm_bo_update later updates the eviction state based on whether the BO is in its preferred domains. In my understanding this includes all cases where the BO is evicted through ttm_bo_evict. Therefore we should apply this definition right from the move callback, not only after amdgpu_vm_bo_update has been called at least once.

Tatsuyuki.

> 
> Regards,
> Christian.
> 
>> 
>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> ---
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>>  3 files changed, 29 insertions(+), 40 deletions(-)
>> 
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> index 7b9762f1cddd..dd6f72e2a1d6 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>   * State for PDs/PTs and per VM BOs which are not at the location they should
>>   * be.
>>   */
>> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
>> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>>  {
>>  	struct amdgpu_vm *vm = vm_bo->vm;
>>  	struct amdgpu_bo *bo = vm_bo->bo;
>>  -	vm_bo->moved = true;
>>  	spin_lock(&vm_bo->vm->status_lock);
>> -	if (bo->tbo.type == ttm_bo_type_kernel)
>> -		list_move(&vm_bo->vm_status, &vm->evicted);
>> -	else
>> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
>> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> +		if (bo->tbo.type == ttm_bo_type_kernel)
>> +			list_move(&vm_bo->eviction_status, &vm->evicted);
>> +		else
>> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
>> +	} else {
>> +		list_del_init(&vm_bo->eviction_status);
>> +	}
>>  	spin_unlock(&vm_bo->vm->status_lock);
>>  }
>> +
>>  /**
>>   * amdgpu_vm_bo_moved - vm_bo is moved
>>   *
>> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  	base->bo = bo;
>>  	base->next = NULL;
>>  	INIT_LIST_HEAD(&base->vm_status);
>> +	INIT_LIST_HEAD(&base->eviction_status);
>>    	if (!bo)
>>  		return;
>> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  	 * is currently evicted. add the bo to the evicted list to make sure it
>>  	 * is validated on next vm use to avoid fault.
>>  	 * */
>> -	amdgpu_vm_bo_evicted(base);
>> +	amdgpu_vm_bo_set_evicted(base, true);
>>  }
>>    /**
>> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  	while (!list_empty(&vm->evicted)) {
>>  		bo_base = list_first_entry(&vm->evicted,
>>  					   struct amdgpu_vm_bo_base,
>> -					   vm_status);
>> +					   eviction_status);
>>  		spin_unlock(&vm->status_lock);
>>    		bo = bo_base->bo;
>> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>  	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>>  		amdgpu_vm_bo_get_memory(bo_va, stats);
>>  -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
>> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>  		amdgpu_vm_bo_get_memory(bo_va, stats);
>>    	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  			return r;
>>  	}
>>  -	/* If the BO is not in its preferred location add it back to
>> -	 * the evicted list so that it gets validated again on the
>> -	 * next command submission.
>> -	 */
>> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> -		uint32_t mem_type = bo->tbo.resource->mem_type;
>> -
>> -		if (!(bo->preferred_domains &
>> -		      amdgpu_mem_type_to_domain(mem_type)))
>> -			amdgpu_vm_bo_evicted(&bo_va->base);
>> -		else
>> -			amdgpu_vm_bo_idle(&bo_va->base);
>> -	} else {
>> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
>> +		amdgpu_vm_bo_idle(&bo_va->base);
>> +	else
>>  		amdgpu_vm_bo_done(&bo_va->base);
>> -	}
>>    	list_splice_init(&bo_va->invalids, &bo_va->valids);
>>  	bo_va->cleared = clear;
>> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>    	spin_lock(&vm->status_lock);
>>  	list_del(&bo_va->base.vm_status);
>> +	list_del(&bo_va->base.eviction_status);
>>  	spin_unlock(&vm->status_lock);
>>    	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
>> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>  	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>>  		bo = bo->parent;
>>  +	/* If the BO is not in its preferred location add it back to
>> +	 * the evicted list so that it gets validated again on the
>> +	 * next command submission.
>> +	 */
>> +	uint32_t mem_type = bo->tbo.resource->mem_type;
>> +	bool suboptimal = !(bo->preferred_domains &
>> +			 amdgpu_mem_type_to_domain(mem_type));
>> +
>>  	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>>  		struct amdgpu_vm *vm = bo_base->vm;
>>  -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>> -			amdgpu_vm_bo_evicted(bo_base);
>> -			continue;
>> -		}
>> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>>    		if (bo_base->moved)
>>  			continue;
>> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>  {
>>  	struct amdgpu_bo_va *bo_va, *tmp;
>>  	u64 total_idle = 0;
>> -	u64 total_evicted = 0;
>>  	u64 total_relocated = 0;
>>  	u64 total_moved = 0;
>>  	u64 total_invalidated = 0;
>>  	u64 total_done = 0;
>>  	unsigned int total_idle_objs = 0;
>> -	unsigned int total_evicted_objs = 0;
>>  	unsigned int total_relocated_objs = 0;
>>  	unsigned int total_moved_objs = 0;
>>  	unsigned int total_invalidated_objs = 0;
>> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>  	total_idle_objs = id;
>>  	id = 0;
>>  -	seq_puts(m, "\tEvicted BOs:\n");
>> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
>> -		if (!bo_va->base.bo)
>> -			continue;
>> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>> -	}
>> -	total_evicted_objs = id;
>> -	id = 0;
>> -
>>  	seq_puts(m, "\tRelocated BOs:\n");
>>  	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>  		if (!bo_va->base.bo)
>> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>    	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>  		   total_idle_objs);
>> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
>> -		   total_evicted_objs);
>>  	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>  		   total_relocated_objs);
>>  	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 204ab13184ed..d9ab97eabda9 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>>    	/* protected by spinlock */
>>  	struct list_head		vm_status;
>> +	struct list_head		eviction_status;
>>    	/* protected by the BO being reserved */
>>  	bool				moved;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> index 96d601e209b8..f78f4040f466 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>>    	spin_lock(&entry->vm->status_lock);
>>  	list_del(&entry->vm_status);
>> +	list_del(&entry->eviction_status);
>>  	spin_unlock(&entry->vm->status_lock);
>>  	amdgpu_bo_unref(&entry->bo);
>>  }
> 

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Christian König]

Am 31.10.23 um 15:39 schrieb Tatsuyuki Ishi:
>
>> On Oct 31, 2023, at 22:55, Christian König <christian.koenig@amd.com> wrote:
>>
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>> In short, eviction never really belonged to the vm_status state machine.
>> I strongly disagree to that.
>>
>>> Even when evicted, the BO could belong to either the moved or done state.
>>> The "evicted" state needed to handle both cases, causing greater confusion.
>>>
>>> Additionally, there were inconsistencies in the definition of an evicted
>>> BO. Some places are based on the `evict` parameter passed from the TTM move
>>> callback, while the others were updated based on whether the BO got its
>>> optimal placement. The second is more accurate for our use case. With this
>>> refactor, the evicted state is solely determined by the second rule.
>> That strongly sounds like you don't understand what the evicted state it good for.
>>
>> The evicted state is for page directories, page tables and per VM BOs which needs to move around before doing the next CS.
>>
>> Please further explain what you try to do here.
> This is mainly an attempt to address inconsistency in the definition of “eviction”. The TTM move callback sets eviction when eviction happens through ttm_bo_evict. This is however not the only way a BO might end up outside its preferred domains.
>
> amdgpu_vm_bo_update later updates the eviction state based on whether the BO is in its preferred domains. In my understanding this includes all cases where the BO is evicted through ttm_bo_evict. Therefore we should apply this definition right from the move callback, not only after amdgpu_vm_bo_update has been called at least once.

No, that is something completely separated. The evicted state just means 
that we need to re-validate the BO.

One cause of this is that TTM moved the BO.

But a different cause is that TTM moved the BO, we tried to validated it 
but fallen back to GTT for now and called amdgpu_vm_bo_update(). 
amdgpu_vm_bo_update() then moves the BO into the evicted state again so 
that we try to move it into VRAM on the next command submission.

This is purely an optimization done to create enough pressure so that 
TTM can do it's work.

Christian.

>
> Tatsuyuki.
>
>> Regards,
>> Christian.
>>
>>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>>>   3 files changed, 29 insertions(+), 40 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index 7b9762f1cddd..dd6f72e2a1d6 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>    * State for PDs/PTs and per VM BOs which are not at the location they should
>>>    * be.
>>>    */
>>> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
>>> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>>>   {
>>>   	struct amdgpu_vm *vm = vm_bo->vm;
>>>   	struct amdgpu_bo *bo = vm_bo->bo;
>>>   -	vm_bo->moved = true;
>>>   	spin_lock(&vm_bo->vm->status_lock);
>>> -	if (bo->tbo.type == ttm_bo_type_kernel)
>>> -		list_move(&vm_bo->vm_status, &vm->evicted);
>>> -	else
>>> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
>>> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> +		if (bo->tbo.type == ttm_bo_type_kernel)
>>> +			list_move(&vm_bo->eviction_status, &vm->evicted);
>>> +		else
>>> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
>>> +	} else {
>>> +		list_del_init(&vm_bo->eviction_status);
>>> +	}
>>>   	spin_unlock(&vm_bo->vm->status_lock);
>>>   }
>>> +
>>>   /**
>>>    * amdgpu_vm_bo_moved - vm_bo is moved
>>>    *
>>> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>>   	base->bo = bo;
>>>   	base->next = NULL;
>>>   	INIT_LIST_HEAD(&base->vm_status);
>>> +	INIT_LIST_HEAD(&base->eviction_status);
>>>     	if (!bo)
>>>   		return;
>>> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>>   	 * is currently evicted. add the bo to the evicted list to make sure it
>>>   	 * is validated on next vm use to avoid fault.
>>>   	 * */
>>> -	amdgpu_vm_bo_evicted(base);
>>> +	amdgpu_vm_bo_set_evicted(base, true);
>>>   }
>>>     /**
>>> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>   	while (!list_empty(&vm->evicted)) {
>>>   		bo_base = list_first_entry(&vm->evicted,
>>>   					   struct amdgpu_vm_bo_base,
>>> -					   vm_status);
>>> +					   eviction_status);
>>>   		spin_unlock(&vm->status_lock);
>>>     		bo = bo_base->bo;
>>> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>>   	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>>>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>>>   -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
>>> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>>> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>>   			return r;
>>>   	}
>>>   -	/* If the BO is not in its preferred location add it back to
>>> -	 * the evicted list so that it gets validated again on the
>>> -	 * next command submission.
>>> -	 */
>>> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> -		uint32_t mem_type = bo->tbo.resource->mem_type;
>>> -
>>> -		if (!(bo->preferred_domains &
>>> -		      amdgpu_mem_type_to_domain(mem_type)))
>>> -			amdgpu_vm_bo_evicted(&bo_va->base);
>>> -		else
>>> -			amdgpu_vm_bo_idle(&bo_va->base);
>>> -	} else {
>>> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
>>> +		amdgpu_vm_bo_idle(&bo_va->base);
>>> +	else
>>>   		amdgpu_vm_bo_done(&bo_va->base);
>>> -	}
>>>     	list_splice_init(&bo_va->invalids, &bo_va->valids);
>>>   	bo_va->cleared = clear;
>>> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>>     	spin_lock(&vm->status_lock);
>>>   	list_del(&bo_va->base.vm_status);
>>> +	list_del(&bo_va->base.eviction_status);
>>>   	spin_unlock(&vm->status_lock);
>>>     	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
>>> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>>   	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>>>   		bo = bo->parent;
>>>   +	/* If the BO is not in its preferred location add it back to
>>> +	 * the evicted list so that it gets validated again on the
>>> +	 * next command submission.
>>> +	 */
>>> +	uint32_t mem_type = bo->tbo.resource->mem_type;
>>> +	bool suboptimal = !(bo->preferred_domains &
>>> +			 amdgpu_mem_type_to_domain(mem_type));
>>> +
>>>   	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>>>   		struct amdgpu_vm *vm = bo_base->vm;
>>>   -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> -			amdgpu_vm_bo_evicted(bo_base);
>>> -			continue;
>>> -		}
>>> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>>>     		if (bo_base->moved)
>>>   			continue;
>>> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>   {
>>>   	struct amdgpu_bo_va *bo_va, *tmp;
>>>   	u64 total_idle = 0;
>>> -	u64 total_evicted = 0;
>>>   	u64 total_relocated = 0;
>>>   	u64 total_moved = 0;
>>>   	u64 total_invalidated = 0;
>>>   	u64 total_done = 0;
>>>   	unsigned int total_idle_objs = 0;
>>> -	unsigned int total_evicted_objs = 0;
>>>   	unsigned int total_relocated_objs = 0;
>>>   	unsigned int total_moved_objs = 0;
>>>   	unsigned int total_invalidated_objs = 0;
>>> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>   	total_idle_objs = id;
>>>   	id = 0;
>>>   -	seq_puts(m, "\tEvicted BOs:\n");
>>> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
>>> -		if (!bo_va->base.bo)
>>> -			continue;
>>> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>>> -	}
>>> -	total_evicted_objs = id;
>>> -	id = 0;
>>> -
>>>   	seq_puts(m, "\tRelocated BOs:\n");
>>>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>>   		if (!bo_va->base.bo)
>>> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>     	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>>   		   total_idle_objs);
>>> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
>>> -		   total_evicted_objs);
>>>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>>   		   total_relocated_objs);
>>>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index 204ab13184ed..d9ab97eabda9 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>>>     	/* protected by spinlock */
>>>   	struct list_head		vm_status;
>>> +	struct list_head		eviction_status;
>>>     	/* protected by the BO being reserved */
>>>   	bool				moved;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> index 96d601e209b8..f78f4040f466 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>>>     	spin_lock(&entry->vm->status_lock);
>>>   	list_del(&entry->vm_status);
>>> +	list_del(&entry->eviction_status);
>>>   	spin_unlock(&entry->vm->status_lock);
>>>   	amdgpu_bo_unref(&entry->bo);
>>>   }

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[Christian König]

Am 31.10.23 um 15:39 schrieb Tatsuyuki Ishi:
>
>> On Oct 31, 2023, at 22:55, Christian König <christian.koenig@amd.com> wrote:
>>
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>> In short, eviction never really belonged to the vm_status state machine.
>> I strongly disagree to that.
>>
>>> Even when evicted, the BO could belong to either the moved or done state.
>>> The "evicted" state needed to handle both cases, causing greater confusion.
>>>
>>> Additionally, there were inconsistencies in the definition of an evicted
>>> BO. Some places are based on the `evict` parameter passed from the TTM move
>>> callback, while the others were updated based on whether the BO got its
>>> optimal placement. The second is more accurate for our use case. With this
>>> refactor, the evicted state is solely determined by the second rule.
>> That strongly sounds like you don't understand what the evicted state it good for.
>>
>> The evicted state is for page directories, page tables and per VM BOs which needs to move around before doing the next CS.
>>
>> Please further explain what you try to do here.
> This is mainly an attempt to address inconsistency in the definition of “eviction”. The TTM move callback sets eviction when eviction happens through ttm_bo_evict. This is however not the only way a BO might end up outside its preferred domains.
>
> amdgpu_vm_bo_update later updates the eviction state based on whether the BO is in its preferred domains. In my understanding this includes all cases where the BO is evicted through ttm_bo_evict. Therefore we should apply this definition right from the move callback, not only after amdgpu_vm_bo_update has been called at least once.

No, that is something completely separated. The evicted state just means 
that we need to re-validate the BO.

One cause of this is that TTM moved the BO.

But a different cause is that TTM moved the BO, we tried to validated it 
but fallen back to GTT for now and called amdgpu_vm_bo_update(). 
amdgpu_vm_bo_update() then moves the BO into the evicted state again so 
that we try to move it into VRAM on the next command submission.

This is purely an optimization done to create enough pressure so that 
TTM can do it's work.

Christian.

>
> Tatsuyuki.
>
>> Regards,
>> Christian.
>>
>>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c    | 67 +++++++++--------------
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h    |  1 +
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c |  1 +
>>>   3 files changed, 29 insertions(+), 40 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index 7b9762f1cddd..dd6f72e2a1d6 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -174,19 +174,23 @@ int amdgpu_vm_set_pasid(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>    * State for PDs/PTs and per VM BOs which are not at the location they should
>>>    * be.
>>>    */
>>> -static void amdgpu_vm_bo_evicted(struct amdgpu_vm_bo_base *vm_bo)
>>> +static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
>>>   {
>>>   	struct amdgpu_vm *vm = vm_bo->vm;
>>>   	struct amdgpu_bo *bo = vm_bo->bo;
>>>   -	vm_bo->moved = true;
>>>   	spin_lock(&vm_bo->vm->status_lock);
>>> -	if (bo->tbo.type == ttm_bo_type_kernel)
>>> -		list_move(&vm_bo->vm_status, &vm->evicted);
>>> -	else
>>> -		list_move_tail(&vm_bo->vm_status, &vm->evicted);
>>> +	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> +		if (bo->tbo.type == ttm_bo_type_kernel)
>>> +			list_move(&vm_bo->eviction_status, &vm->evicted);
>>> +		else
>>> +			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
>>> +	} else {
>>> +		list_del_init(&vm_bo->eviction_status);
>>> +	}
>>>   	spin_unlock(&vm_bo->vm->status_lock);
>>>   }
>>> +
>>>   /**
>>>    * amdgpu_vm_bo_moved - vm_bo is moved
>>>    *
>>> @@ -310,6 +314,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>>   	base->bo = bo;
>>>   	base->next = NULL;
>>>   	INIT_LIST_HEAD(&base->vm_status);
>>> +	INIT_LIST_HEAD(&base->eviction_status);
>>>     	if (!bo)
>>>   		return;
>>> @@ -336,7 +341,7 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>>   	 * is currently evicted. add the bo to the evicted list to make sure it
>>>   	 * is validated on next vm use to avoid fault.
>>>   	 * */
>>> -	amdgpu_vm_bo_evicted(base);
>>> +	amdgpu_vm_bo_set_evicted(base, true);
>>>   }
>>>     /**
>>> @@ -460,7 +465,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>>   	while (!list_empty(&vm->evicted)) {
>>>   		bo_base = list_first_entry(&vm->evicted,
>>>   					   struct amdgpu_vm_bo_base,
>>> -					   vm_status);
>>> +					   eviction_status);
>>>   		spin_unlock(&vm->status_lock);
>>>     		bo = bo_base->bo;
>>> @@ -1034,7 +1039,7 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>>   	list_for_each_entry_safe(bo_va, tmp, &vm->idle, base.vm_status)
>>>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>>>   -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status)
>>> +	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>>> @@ -1153,21 +1158,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>>   			return r;
>>>   	}
>>>   -	/* If the BO is not in its preferred location add it back to
>>> -	 * the evicted list so that it gets validated again on the
>>> -	 * next command submission.
>>> -	 */
>>> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> -		uint32_t mem_type = bo->tbo.resource->mem_type;
>>> -
>>> -		if (!(bo->preferred_domains &
>>> -		      amdgpu_mem_type_to_domain(mem_type)))
>>> -			amdgpu_vm_bo_evicted(&bo_va->base);
>>> -		else
>>> -			amdgpu_vm_bo_idle(&bo_va->base);
>>> -	} else {
>>> +	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv)
>>> +		amdgpu_vm_bo_idle(&bo_va->base);
>>> +	else
>>>   		amdgpu_vm_bo_done(&bo_va->base);
>>> -	}
>>>     	list_splice_init(&bo_va->invalids, &bo_va->valids);
>>>   	bo_va->cleared = clear;
>>> @@ -1883,6 +1877,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>>     	spin_lock(&vm->status_lock);
>>>   	list_del(&bo_va->base.vm_status);
>>> +	list_del(&bo_va->base.eviction_status);
>>>   	spin_unlock(&vm->status_lock);
>>>     	list_for_each_entry_safe(mapping, next, &bo_va->valids, list) {
>>> @@ -1959,13 +1954,18 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>>   	if (bo->parent && (amdgpu_bo_shadowed(bo->parent) == bo))
>>>   		bo = bo->parent;
>>>   +	/* If the BO is not in its preferred location add it back to
>>> +	 * the evicted list so that it gets validated again on the
>>> +	 * next command submission.
>>> +	 */
>>> +	uint32_t mem_type = bo->tbo.resource->mem_type;
>>> +	bool suboptimal = !(bo->preferred_domains &
>>> +			 amdgpu_mem_type_to_domain(mem_type));
>>> +
>>>   	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
>>>   		struct amdgpu_vm *vm = bo_base->vm;
>>>   -		if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
>>> -			amdgpu_vm_bo_evicted(bo_base);
>>> -			continue;
>>> -		}
>>> +		amdgpu_vm_bo_set_evicted(bo_base, suboptimal);
>>>     		if (bo_base->moved)
>>>   			continue;
>>> @@ -2648,13 +2648,11 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>   {
>>>   	struct amdgpu_bo_va *bo_va, *tmp;
>>>   	u64 total_idle = 0;
>>> -	u64 total_evicted = 0;
>>>   	u64 total_relocated = 0;
>>>   	u64 total_moved = 0;
>>>   	u64 total_invalidated = 0;
>>>   	u64 total_done = 0;
>>>   	unsigned int total_idle_objs = 0;
>>> -	unsigned int total_evicted_objs = 0;
>>>   	unsigned int total_relocated_objs = 0;
>>>   	unsigned int total_moved_objs = 0;
>>>   	unsigned int total_invalidated_objs = 0;
>>> @@ -2671,15 +2669,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>   	total_idle_objs = id;
>>>   	id = 0;
>>>   -	seq_puts(m, "\tEvicted BOs:\n");
>>> -	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.vm_status) {
>>> -		if (!bo_va->base.bo)
>>> -			continue;
>>> -		total_evicted += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>>> -	}
>>> -	total_evicted_objs = id;
>>> -	id = 0;
>>> -
>>>   	seq_puts(m, "\tRelocated BOs:\n");
>>>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>>   		if (!bo_va->base.bo)
>>> @@ -2718,8 +2707,6 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>>     	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>>   		   total_idle_objs);
>>> -	seq_printf(m, "\tTotal evicted size:     %12lld\tobjs:\t%d\n", total_evicted,
>>> -		   total_evicted_objs);
>>>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>>   		   total_relocated_objs);
>>>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index 204ab13184ed..d9ab97eabda9 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -156,6 +156,7 @@ struct amdgpu_vm_bo_base {
>>>     	/* protected by spinlock */
>>>   	struct list_head		vm_status;
>>> +	struct list_head		eviction_status;
>>>     	/* protected by the BO being reserved */
>>>   	bool				moved;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> index 96d601e209b8..f78f4040f466 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
>>> @@ -652,6 +652,7 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
>>>     	spin_lock(&entry->vm->status_lock);
>>>   	list_del(&entry->vm_status);
>>> +	list_del(&entry->eviction_status);
>>>   	spin_unlock(&entry->vm->status_lock);
>>>   	amdgpu_bo_unref(&entry->bo);
>>>   }

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next drm-intel/for-linux-next-fixes drm-tip/drm-tip linus/master v6.6 next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-3-ishitatsuyuki%40gmail.com
patch subject: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311010709.XbwKjVaq-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311010709.XbwKjVaq-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311010709.XbwKjVaq-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: Function parameter or member 'evicted' not described in 'amdgpu_vm_bo_set_evicted'
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: expecting prototype for amdgpu_vm_bo_evicted(). Prototype was for amdgpu_vm_bo_set_evicted() instead


vim +178 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

dcb388eddb5f1b Nirmoy Das      2021-06-28  168  
bcdc9fd634d1f0 Christian König 2018-08-30  169  /**
bcdc9fd634d1f0 Christian König 2018-08-30  170   * amdgpu_vm_bo_evicted - vm_bo is evicted
bcdc9fd634d1f0 Christian König 2018-08-30  171   *
bcdc9fd634d1f0 Christian König 2018-08-30  172   * @vm_bo: vm_bo which is evicted
bcdc9fd634d1f0 Christian König 2018-08-30  173   *
bcdc9fd634d1f0 Christian König 2018-08-30  174   * State for PDs/PTs and per VM BOs which are not at the location they should
bcdc9fd634d1f0 Christian König 2018-08-30  175   * be.
bcdc9fd634d1f0 Christian König 2018-08-30  176   */
cac82290238e47 Tatsuyuki Ishi  2023-10-31  177  static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
bcdc9fd634d1f0 Christian König 2018-08-30 @178  {
bcdc9fd634d1f0 Christian König 2018-08-30  179  	struct amdgpu_vm *vm = vm_bo->vm;
bcdc9fd634d1f0 Christian König 2018-08-30  180  	struct amdgpu_bo *bo = vm_bo->bo;
bcdc9fd634d1f0 Christian König 2018-08-30  181  
757eb2bedd08a1 Philip Yang     2022-09-15  182  	spin_lock(&vm_bo->vm->status_lock);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  183  	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
bcdc9fd634d1f0 Christian König 2018-08-30  184  		if (bo->tbo.type == ttm_bo_type_kernel)
cac82290238e47 Tatsuyuki Ishi  2023-10-31  185  			list_move(&vm_bo->eviction_status, &vm->evicted);
bcdc9fd634d1f0 Christian König 2018-08-30  186  		else
cac82290238e47 Tatsuyuki Ishi  2023-10-31  187  			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  188  	} else {
cac82290238e47 Tatsuyuki Ishi  2023-10-31  189  		list_del_init(&vm_bo->eviction_status);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  190  	}
757eb2bedd08a1 Philip Yang     2022-09-15  191  	spin_unlock(&vm_bo->vm->status_lock);
bcdc9fd634d1f0 Christian König 2018-08-30  192  }
cac82290238e47 Tatsuyuki Ishi  2023-10-31  193  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next drm-intel/for-linux-next-fixes drm-tip/drm-tip linus/master v6.6 next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-3-ishitatsuyuki%40gmail.com
patch subject: [PATCH 2/6] drm/amdgpu: Separate eviction from VM status.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311010709.XbwKjVaq-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311010709.XbwKjVaq-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311010709.XbwKjVaq-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: Function parameter or member 'evicted' not described in 'amdgpu_vm_bo_set_evicted'
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: expecting prototype for amdgpu_vm_bo_evicted(). Prototype was for amdgpu_vm_bo_set_evicted() instead


vim +178 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

dcb388eddb5f1b Nirmoy Das      2021-06-28  168  
bcdc9fd634d1f0 Christian König 2018-08-30  169  /**
bcdc9fd634d1f0 Christian König 2018-08-30  170   * amdgpu_vm_bo_evicted - vm_bo is evicted
bcdc9fd634d1f0 Christian König 2018-08-30  171   *
bcdc9fd634d1f0 Christian König 2018-08-30  172   * @vm_bo: vm_bo which is evicted
bcdc9fd634d1f0 Christian König 2018-08-30  173   *
bcdc9fd634d1f0 Christian König 2018-08-30  174   * State for PDs/PTs and per VM BOs which are not at the location they should
bcdc9fd634d1f0 Christian König 2018-08-30  175   * be.
bcdc9fd634d1f0 Christian König 2018-08-30  176   */
cac82290238e47 Tatsuyuki Ishi  2023-10-31  177  static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evicted)
bcdc9fd634d1f0 Christian König 2018-08-30 @178  {
bcdc9fd634d1f0 Christian König 2018-08-30  179  	struct amdgpu_vm *vm = vm_bo->vm;
bcdc9fd634d1f0 Christian König 2018-08-30  180  	struct amdgpu_bo *bo = vm_bo->bo;
bcdc9fd634d1f0 Christian König 2018-08-30  181  
757eb2bedd08a1 Philip Yang     2022-09-15  182  	spin_lock(&vm_bo->vm->status_lock);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  183  	if (evicted && bo->tbo.base.resv == vm->root.bo->tbo.base.resv) {
bcdc9fd634d1f0 Christian König 2018-08-30  184  		if (bo->tbo.type == ttm_bo_type_kernel)
cac82290238e47 Tatsuyuki Ishi  2023-10-31  185  			list_move(&vm_bo->eviction_status, &vm->evicted);
bcdc9fd634d1f0 Christian König 2018-08-30  186  		else
cac82290238e47 Tatsuyuki Ishi  2023-10-31  187  			list_move_tail(&vm_bo->eviction_status, &vm->evicted);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  188  	} else {
cac82290238e47 Tatsuyuki Ishi  2023-10-31  189  		list_del_init(&vm_bo->eviction_status);
cac82290238e47 Tatsuyuki Ishi  2023-10-31  190  	}
757eb2bedd08a1 Philip Yang     2022-09-15  191  	spin_unlock(&vm_bo->vm->status_lock);
bcdc9fd634d1f0 Christian König 2018-08-30  192  }
cac82290238e47 Tatsuyuki Ishi  2023-10-31  193  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

The current amdgpu_gem_va_update_vm only tries to perform updates for the
BO specified in the GEM ioctl; however, when a binding is split, the
adjacent bindings also need to be updated. Such updates currently ends up
getting deferred until next submission which causes stalls.

Introduce a new state "dirty", shared between per-VM BOs and traditional
BOs, containing all BOs that have pending updates in `invalids`.
amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
in the dirty state.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
 3 files changed, 63 insertions(+), 24 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index a1b15d0d6c48..01d3a97248b0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
  * vital here, so they are not reported back to userspace.
  */
 static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
-				    struct amdgpu_vm *vm,
-				    struct amdgpu_bo_va *bo_va,
-				    uint32_t operation)
+				    struct amdgpu_vm *vm)
 {
+	struct amdgpu_bo_va *bo_va;
 	int r;
 
 	if (!amdgpu_vm_ready(vm))
@@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
 	if (r)
 		goto error;
 
-	if (operation == AMDGPU_VA_OP_MAP ||
-	    operation == AMDGPU_VA_OP_REPLACE) {
+	spin_lock(&vm->status_lock);
+	while (!list_empty(&vm->dirty)) {
+		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
+					 base.vm_status);
+		spin_unlock(&vm->status_lock);
+
 		r = amdgpu_vm_bo_update(adev, bo_va, false);
 		if (r)
 			goto error;
+		spin_lock(&vm->status_lock);
 	}
+	spin_unlock(&vm->status_lock);
 
 	r = amdgpu_vm_update_pdes(adev, vm, false);
 
@@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		break;
 	}
 	if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
-		amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
-					args->operation);
+		amdgpu_gem_va_update_vm(adev, &fpriv->vm);
 
 error:
 	drm_exec_fini(&exec);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dd6f72e2a1d6..01d31891cd05 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
 	spin_unlock(&vm_bo->vm->status_lock);
 }
 
+/**
+ * amdgpu_vm_bo_dirty - vm_bo is dirty
+ *
+ * @vm_bo: vm_bo which is dirty
+ *
+ * State for normal and per VM BOs that are not moved, but have new entries in
+ * bo_va->invalids.
+ */
+static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
+{
+	spin_lock(&vm_bo->vm->status_lock);
+	list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
+	spin_unlock(&vm_bo->vm->status_lock);
+}
+
 /**
  * amdgpu_vm_bo_moved - vm_bo is moved
  *
@@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
 	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
+	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
+		amdgpu_vm_bo_get_memory(bo_va, stats);
+
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
@@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
 			dma_resv_unlock(resv);
 		spin_lock(&vm->status_lock);
 	}
+
+	while (!list_empty(&vm->dirty)) {
+		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
+					 base.vm_status);
+		spin_unlock(&vm->status_lock);
+
+		r = amdgpu_vm_bo_update(adev, bo_va, false);
+		if (r)
+			return r;
+		spin_lock(&vm->status_lock);
+	}
 	spin_unlock(&vm->status_lock);
 
 	return 0;
@@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
 				    struct amdgpu_bo_va_mapping *mapping)
 {
 	struct amdgpu_vm *vm = bo_va->base.vm;
-	struct amdgpu_bo *bo = bo_va->base.bo;
 
 	mapping->bo_va = bo_va;
 	list_add(&mapping->list, &bo_va->invalids);
 	amdgpu_vm_it_insert(mapping, &vm->va);
+	if (!bo_va->base.moved)
+		amdgpu_vm_bo_dirty(&bo_va->base);
 
 	if (mapping->flags & AMDGPU_PTE_PRT)
 		amdgpu_vm_prt_get(adev);
 
-	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-	    !bo_va->base.moved) {
-		amdgpu_vm_bo_moved(&bo_va->base);
-	}
 	trace_amdgpu_vm_bo_map(bo_va, mapping);
 }
 
@@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 			before->flags = tmp->flags;
 			before->bo_va = tmp->bo_va;
 			list_add(&before->list, &tmp->bo_va->invalids);
+			if (!tmp->bo_va->base.moved)
+				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
 		}
 
 		/* Remember mapping split at the end */
@@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 			after->flags = tmp->flags;
 			after->bo_va = tmp->bo_va;
 			list_add(&after->list, &tmp->bo_va->invalids);
+			if (!tmp->bo_va->base.moved)
+				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
 		}
 
 		list_del(&tmp->list);
@@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 
 	/* Insert partial mapping before the range */
 	if (!list_empty(&before->list)) {
-		struct amdgpu_bo *bo = before->bo_va->base.bo;
-
 		amdgpu_vm_it_insert(before, &vm->va);
 		if (before->flags & AMDGPU_PTE_PRT)
 			amdgpu_vm_prt_get(adev);
-
-		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-		    !before->bo_va->base.moved)
-			amdgpu_vm_bo_moved(&before->bo_va->base);
 	} else {
 		kfree(before);
 	}
 
 	/* Insert partial mapping after the range */
 	if (!list_empty(&after->list)) {
-		struct amdgpu_bo *bo = after->bo_va->base.bo;
-
 		amdgpu_vm_it_insert(after, &vm->va);
 		if (after->flags & AMDGPU_PTE_PRT)
 			amdgpu_vm_prt_get(adev);
-
-		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-		    !after->bo_va->base.moved)
-			amdgpu_vm_bo_moved(&after->bo_va->base);
 	} else {
 		kfree(after);
 	}
@@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
 	INIT_LIST_HEAD(&vm->evicted);
 	INIT_LIST_HEAD(&vm->relocated);
 	INIT_LIST_HEAD(&vm->moved);
+	INIT_LIST_HEAD(&vm->dirty);
 	INIT_LIST_HEAD(&vm->idle);
 	INIT_LIST_HEAD(&vm->invalidated);
 	spin_lock_init(&vm->status_lock);
@@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 {
 	struct amdgpu_bo_va *bo_va, *tmp;
 	u64 total_idle = 0;
+	u64 total_dirty = 0;
 	u64 total_relocated = 0;
 	u64 total_moved = 0;
 	u64 total_invalidated = 0;
 	u64 total_done = 0;
 	unsigned int total_idle_objs = 0;
+	unsigned int total_dirty_objs = 0;
 	unsigned int total_relocated_objs = 0;
 	unsigned int total_moved_objs = 0;
 	unsigned int total_invalidated_objs = 0;
@@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 	total_idle_objs = id;
 	id = 0;
 
+	seq_puts(m, "\tDirty BOs:\n");
+	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
+		if (!bo_va->base.bo)
+			continue;
+		total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
+	}
+	total_dirty_objs = id;
+	id = 0;
+
 	seq_puts(m, "\tRelocated BOs:\n");
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
 		if (!bo_va->base.bo)
@@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 
 	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
 		   total_idle_objs);
+	seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
+		   total_dirty_objs);
 	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
 		   total_relocated_objs);
 	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index d9ab97eabda9..f91d4fcf80b8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -276,6 +276,9 @@ struct amdgpu_vm {
 	/* per VM BOs moved, but not yet updated in the PT */
 	struct list_head	moved;
 
+	/* normal and per VM BOs that are not moved, but have new PT entries */
+	struct list_head	dirty;
+
 	/* All BOs of this VM not currently in the state machine */
 	struct list_head	idle;
 
-- 
2.42.0

[PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

The current amdgpu_gem_va_update_vm only tries to perform updates for the
BO specified in the GEM ioctl; however, when a binding is split, the
adjacent bindings also need to be updated. Such updates currently ends up
getting deferred until next submission which causes stalls.

Introduce a new state "dirty", shared between per-VM BOs and traditional
BOs, containing all BOs that have pending updates in `invalids`.
amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
in the dirty state.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
 3 files changed, 63 insertions(+), 24 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index a1b15d0d6c48..01d3a97248b0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
  * vital here, so they are not reported back to userspace.
  */
 static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
-				    struct amdgpu_vm *vm,
-				    struct amdgpu_bo_va *bo_va,
-				    uint32_t operation)
+				    struct amdgpu_vm *vm)
 {
+	struct amdgpu_bo_va *bo_va;
 	int r;
 
 	if (!amdgpu_vm_ready(vm))
@@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
 	if (r)
 		goto error;
 
-	if (operation == AMDGPU_VA_OP_MAP ||
-	    operation == AMDGPU_VA_OP_REPLACE) {
+	spin_lock(&vm->status_lock);
+	while (!list_empty(&vm->dirty)) {
+		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
+					 base.vm_status);
+		spin_unlock(&vm->status_lock);
+
 		r = amdgpu_vm_bo_update(adev, bo_va, false);
 		if (r)
 			goto error;
+		spin_lock(&vm->status_lock);
 	}
+	spin_unlock(&vm->status_lock);
 
 	r = amdgpu_vm_update_pdes(adev, vm, false);
 
@@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		break;
 	}
 	if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
-		amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
-					args->operation);
+		amdgpu_gem_va_update_vm(adev, &fpriv->vm);
 
 error:
 	drm_exec_fini(&exec);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dd6f72e2a1d6..01d31891cd05 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
 	spin_unlock(&vm_bo->vm->status_lock);
 }
 
+/**
+ * amdgpu_vm_bo_dirty - vm_bo is dirty
+ *
+ * @vm_bo: vm_bo which is dirty
+ *
+ * State for normal and per VM BOs that are not moved, but have new entries in
+ * bo_va->invalids.
+ */
+static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
+{
+	spin_lock(&vm_bo->vm->status_lock);
+	list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
+	spin_unlock(&vm_bo->vm->status_lock);
+}
+
 /**
  * amdgpu_vm_bo_moved - vm_bo is moved
  *
@@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
 	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
+	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
+		amdgpu_vm_bo_get_memory(bo_va, stats);
+
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
 		amdgpu_vm_bo_get_memory(bo_va, stats);
 
@@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
 			dma_resv_unlock(resv);
 		spin_lock(&vm->status_lock);
 	}
+
+	while (!list_empty(&vm->dirty)) {
+		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
+					 base.vm_status);
+		spin_unlock(&vm->status_lock);
+
+		r = amdgpu_vm_bo_update(adev, bo_va, false);
+		if (r)
+			return r;
+		spin_lock(&vm->status_lock);
+	}
 	spin_unlock(&vm->status_lock);
 
 	return 0;
@@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
 				    struct amdgpu_bo_va_mapping *mapping)
 {
 	struct amdgpu_vm *vm = bo_va->base.vm;
-	struct amdgpu_bo *bo = bo_va->base.bo;
 
 	mapping->bo_va = bo_va;
 	list_add(&mapping->list, &bo_va->invalids);
 	amdgpu_vm_it_insert(mapping, &vm->va);
+	if (!bo_va->base.moved)
+		amdgpu_vm_bo_dirty(&bo_va->base);
 
 	if (mapping->flags & AMDGPU_PTE_PRT)
 		amdgpu_vm_prt_get(adev);
 
-	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-	    !bo_va->base.moved) {
-		amdgpu_vm_bo_moved(&bo_va->base);
-	}
 	trace_amdgpu_vm_bo_map(bo_va, mapping);
 }
 
@@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 			before->flags = tmp->flags;
 			before->bo_va = tmp->bo_va;
 			list_add(&before->list, &tmp->bo_va->invalids);
+			if (!tmp->bo_va->base.moved)
+				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
 		}
 
 		/* Remember mapping split at the end */
@@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 			after->flags = tmp->flags;
 			after->bo_va = tmp->bo_va;
 			list_add(&after->list, &tmp->bo_va->invalids);
+			if (!tmp->bo_va->base.moved)
+				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
 		}
 
 		list_del(&tmp->list);
@@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 
 	/* Insert partial mapping before the range */
 	if (!list_empty(&before->list)) {
-		struct amdgpu_bo *bo = before->bo_va->base.bo;
-
 		amdgpu_vm_it_insert(before, &vm->va);
 		if (before->flags & AMDGPU_PTE_PRT)
 			amdgpu_vm_prt_get(adev);
-
-		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-		    !before->bo_va->base.moved)
-			amdgpu_vm_bo_moved(&before->bo_va->base);
 	} else {
 		kfree(before);
 	}
 
 	/* Insert partial mapping after the range */
 	if (!list_empty(&after->list)) {
-		struct amdgpu_bo *bo = after->bo_va->base.bo;
-
 		amdgpu_vm_it_insert(after, &vm->va);
 		if (after->flags & AMDGPU_PTE_PRT)
 			amdgpu_vm_prt_get(adev);
-
-		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
-		    !after->bo_va->base.moved)
-			amdgpu_vm_bo_moved(&after->bo_va->base);
 	} else {
 		kfree(after);
 	}
@@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
 	INIT_LIST_HEAD(&vm->evicted);
 	INIT_LIST_HEAD(&vm->relocated);
 	INIT_LIST_HEAD(&vm->moved);
+	INIT_LIST_HEAD(&vm->dirty);
 	INIT_LIST_HEAD(&vm->idle);
 	INIT_LIST_HEAD(&vm->invalidated);
 	spin_lock_init(&vm->status_lock);
@@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 {
 	struct amdgpu_bo_va *bo_va, *tmp;
 	u64 total_idle = 0;
+	u64 total_dirty = 0;
 	u64 total_relocated = 0;
 	u64 total_moved = 0;
 	u64 total_invalidated = 0;
 	u64 total_done = 0;
 	unsigned int total_idle_objs = 0;
+	unsigned int total_dirty_objs = 0;
 	unsigned int total_relocated_objs = 0;
 	unsigned int total_moved_objs = 0;
 	unsigned int total_invalidated_objs = 0;
@@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 	total_idle_objs = id;
 	id = 0;
 
+	seq_puts(m, "\tDirty BOs:\n");
+	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
+		if (!bo_va->base.bo)
+			continue;
+		total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
+	}
+	total_dirty_objs = id;
+	id = 0;
+
 	seq_puts(m, "\tRelocated BOs:\n");
 	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
 		if (!bo_va->base.bo)
@@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
 
 	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
 		   total_idle_objs);
+	seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
+		   total_dirty_objs);
 	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
 		   total_relocated_objs);
 	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index d9ab97eabda9..f91d4fcf80b8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -276,6 +276,9 @@ struct amdgpu_vm {
 	/* per VM BOs moved, but not yet updated in the PT */
 	struct list_head	moved;
 
+	/* normal and per VM BOs that are not moved, but have new PT entries */
+	struct list_head	dirty;
+
 	/* All BOs of this VM not currently in the state machine */
 	struct list_head	idle;
 
-- 
2.42.0

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> The current amdgpu_gem_va_update_vm only tries to perform updates for the
> BO specified in the GEM ioctl; however, when a binding is split, the
> adjacent bindings also need to be updated. Such updates currently ends up
> getting deferred until next submission which causes stalls.

Yeah, that is a necessity. The hardware simply doesn't support what you 
try to do here in all cases.

So this approach won't work in general.

Regards,
Christian.

>
> Introduce a new state "dirty", shared between per-VM BOs and traditional
> BOs, containing all BOs that have pending updates in `invalids`.
> amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
> in the dirty state.
>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>   3 files changed, 63 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index a1b15d0d6c48..01d3a97248b0 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>    * vital here, so they are not reported back to userspace.
>    */
>   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> -				    struct amdgpu_vm *vm,
> -				    struct amdgpu_bo_va *bo_va,
> -				    uint32_t operation)
> +				    struct amdgpu_vm *vm)
>   {
> +	struct amdgpu_bo_va *bo_va;
>   	int r;
>   
>   	if (!amdgpu_vm_ready(vm))
> @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>   	if (r)
>   		goto error;
>   
> -	if (operation == AMDGPU_VA_OP_MAP ||
> -	    operation == AMDGPU_VA_OP_REPLACE) {
> +	spin_lock(&vm->status_lock);
> +	while (!list_empty(&vm->dirty)) {
> +		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> +					 base.vm_status);
> +		spin_unlock(&vm->status_lock);
> +
>   		r = amdgpu_vm_bo_update(adev, bo_va, false);
>   		if (r)
>   			goto error;
> +		spin_lock(&vm->status_lock);
>   	}
> +	spin_unlock(&vm->status_lock);
>   
>   	r = amdgpu_vm_update_pdes(adev, vm, false);
>   
> @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   		break;
>   	}
>   	if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
> -		amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> -					args->operation);
> +		amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>   
>   error:
>   	drm_exec_fini(&exec);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index dd6f72e2a1d6..01d31891cd05 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>   	spin_unlock(&vm_bo->vm->status_lock);
>   }
>   
> +/**
> + * amdgpu_vm_bo_dirty - vm_bo is dirty
> + *
> + * @vm_bo: vm_bo which is dirty
> + *
> + * State for normal and per VM BOs that are not moved, but have new entries in
> + * bo_va->invalids.
> + */
> +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> +{
> +	spin_lock(&vm_bo->vm->status_lock);
> +	list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> +	spin_unlock(&vm_bo->vm->status_lock);
> +}
> +
>   /**
>    * amdgpu_vm_bo_moved - vm_bo is moved
>    *
> @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>   	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> +	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
> +		amdgpu_vm_bo_get_memory(bo_va, stats);
> +
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>   			dma_resv_unlock(resv);
>   		spin_lock(&vm->status_lock);
>   	}
> +
> +	while (!list_empty(&vm->dirty)) {
> +		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> +					 base.vm_status);
> +		spin_unlock(&vm->status_lock);
> +
> +		r = amdgpu_vm_bo_update(adev, bo_va, false);
> +		if (r)
> +			return r;
> +		spin_lock(&vm->status_lock);
> +	}
>   	spin_unlock(&vm->status_lock);
>   
>   	return 0;
> @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>   				    struct amdgpu_bo_va_mapping *mapping)
>   {
>   	struct amdgpu_vm *vm = bo_va->base.vm;
> -	struct amdgpu_bo *bo = bo_va->base.bo;
>   
>   	mapping->bo_va = bo_va;
>   	list_add(&mapping->list, &bo_va->invalids);
>   	amdgpu_vm_it_insert(mapping, &vm->va);
> +	if (!bo_va->base.moved)
> +		amdgpu_vm_bo_dirty(&bo_va->base);
>   
>   	if (mapping->flags & AMDGPU_PTE_PRT)
>   		amdgpu_vm_prt_get(adev);
>   
> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -	    !bo_va->base.moved) {
> -		amdgpu_vm_bo_moved(&bo_va->base);
> -	}
>   	trace_amdgpu_vm_bo_map(bo_va, mapping);
>   }
>   
> @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   			before->flags = tmp->flags;
>   			before->bo_va = tmp->bo_va;
>   			list_add(&before->list, &tmp->bo_va->invalids);
> +			if (!tmp->bo_va->base.moved)
> +				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>   		}
>   
>   		/* Remember mapping split at the end */
> @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   			after->flags = tmp->flags;
>   			after->bo_va = tmp->bo_va;
>   			list_add(&after->list, &tmp->bo_va->invalids);
> +			if (!tmp->bo_va->base.moved)
> +				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>   		}
>   
>   		list_del(&tmp->list);
> @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   
>   	/* Insert partial mapping before the range */
>   	if (!list_empty(&before->list)) {
> -		struct amdgpu_bo *bo = before->bo_va->base.bo;
> -
>   		amdgpu_vm_it_insert(before, &vm->va);
>   		if (before->flags & AMDGPU_PTE_PRT)
>   			amdgpu_vm_prt_get(adev);
> -
> -		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -		    !before->bo_va->base.moved)
> -			amdgpu_vm_bo_moved(&before->bo_va->base);
>   	} else {
>   		kfree(before);
>   	}
>   
>   	/* Insert partial mapping after the range */
>   	if (!list_empty(&after->list)) {
> -		struct amdgpu_bo *bo = after->bo_va->base.bo;
> -
>   		amdgpu_vm_it_insert(after, &vm->va);
>   		if (after->flags & AMDGPU_PTE_PRT)
>   			amdgpu_vm_prt_get(adev);
> -
> -		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -		    !after->bo_va->base.moved)
> -			amdgpu_vm_bo_moved(&after->bo_va->base);
>   	} else {
>   		kfree(after);
>   	}
> @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>   	INIT_LIST_HEAD(&vm->evicted);
>   	INIT_LIST_HEAD(&vm->relocated);
>   	INIT_LIST_HEAD(&vm->moved);
> +	INIT_LIST_HEAD(&vm->dirty);
>   	INIT_LIST_HEAD(&vm->idle);
>   	INIT_LIST_HEAD(&vm->invalidated);
>   	spin_lock_init(&vm->status_lock);
> @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   {
>   	struct amdgpu_bo_va *bo_va, *tmp;
>   	u64 total_idle = 0;
> +	u64 total_dirty = 0;
>   	u64 total_relocated = 0;
>   	u64 total_moved = 0;
>   	u64 total_invalidated = 0;
>   	u64 total_done = 0;
>   	unsigned int total_idle_objs = 0;
> +	unsigned int total_dirty_objs = 0;
>   	unsigned int total_relocated_objs = 0;
>   	unsigned int total_moved_objs = 0;
>   	unsigned int total_invalidated_objs = 0;
> @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   	total_idle_objs = id;
>   	id = 0;
>   
> +	seq_puts(m, "\tDirty BOs:\n");
> +	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
> +		if (!bo_va->base.bo)
> +			continue;
> +		total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
> +	}
> +	total_dirty_objs = id;
> +	id = 0;
> +
>   	seq_puts(m, "\tRelocated BOs:\n");
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>   		if (!bo_va->base.bo)
> @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   
>   	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>   		   total_idle_objs);
> +	seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
> +		   total_dirty_objs);
>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>   		   total_relocated_objs);
>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index d9ab97eabda9..f91d4fcf80b8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -276,6 +276,9 @@ struct amdgpu_vm {
>   	/* per VM BOs moved, but not yet updated in the PT */
>   	struct list_head	moved;
>   
> +	/* normal and per VM BOs that are not moved, but have new PT entries */
> +	struct list_head	dirty;
> +
>   	/* All BOs of this VM not currently in the state machine */
>   	struct list_head	idle;
>   

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> The current amdgpu_gem_va_update_vm only tries to perform updates for the
> BO specified in the GEM ioctl; however, when a binding is split, the
> adjacent bindings also need to be updated. Such updates currently ends up
> getting deferred until next submission which causes stalls.

Yeah, that is a necessity. The hardware simply doesn't support what you 
try to do here in all cases.

So this approach won't work in general.

Regards,
Christian.

>
> Introduce a new state "dirty", shared between per-VM BOs and traditional
> BOs, containing all BOs that have pending updates in `invalids`.
> amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
> in the dirty state.
>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>   3 files changed, 63 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index a1b15d0d6c48..01d3a97248b0 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>    * vital here, so they are not reported back to userspace.
>    */
>   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> -				    struct amdgpu_vm *vm,
> -				    struct amdgpu_bo_va *bo_va,
> -				    uint32_t operation)
> +				    struct amdgpu_vm *vm)
>   {
> +	struct amdgpu_bo_va *bo_va;
>   	int r;
>   
>   	if (!amdgpu_vm_ready(vm))
> @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>   	if (r)
>   		goto error;
>   
> -	if (operation == AMDGPU_VA_OP_MAP ||
> -	    operation == AMDGPU_VA_OP_REPLACE) {
> +	spin_lock(&vm->status_lock);
> +	while (!list_empty(&vm->dirty)) {
> +		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> +					 base.vm_status);
> +		spin_unlock(&vm->status_lock);
> +
>   		r = amdgpu_vm_bo_update(adev, bo_va, false);
>   		if (r)
>   			goto error;
> +		spin_lock(&vm->status_lock);
>   	}
> +	spin_unlock(&vm->status_lock);
>   
>   	r = amdgpu_vm_update_pdes(adev, vm, false);
>   
> @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   		break;
>   	}
>   	if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
> -		amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> -					args->operation);
> +		amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>   
>   error:
>   	drm_exec_fini(&exec);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index dd6f72e2a1d6..01d31891cd05 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>   	spin_unlock(&vm_bo->vm->status_lock);
>   }
>   
> +/**
> + * amdgpu_vm_bo_dirty - vm_bo is dirty
> + *
> + * @vm_bo: vm_bo which is dirty
> + *
> + * State for normal and per VM BOs that are not moved, but have new entries in
> + * bo_va->invalids.
> + */
> +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> +{
> +	spin_lock(&vm_bo->vm->status_lock);
> +	list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> +	spin_unlock(&vm_bo->vm->status_lock);
> +}
> +
>   /**
>    * amdgpu_vm_bo_moved - vm_bo is moved
>    *
> @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>   	list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> +	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
> +		amdgpu_vm_bo_get_memory(bo_va, stats);
> +
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>   		amdgpu_vm_bo_get_memory(bo_va, stats);
>   
> @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>   			dma_resv_unlock(resv);
>   		spin_lock(&vm->status_lock);
>   	}
> +
> +	while (!list_empty(&vm->dirty)) {
> +		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> +					 base.vm_status);
> +		spin_unlock(&vm->status_lock);
> +
> +		r = amdgpu_vm_bo_update(adev, bo_va, false);
> +		if (r)
> +			return r;
> +		spin_lock(&vm->status_lock);
> +	}
>   	spin_unlock(&vm->status_lock);
>   
>   	return 0;
> @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>   				    struct amdgpu_bo_va_mapping *mapping)
>   {
>   	struct amdgpu_vm *vm = bo_va->base.vm;
> -	struct amdgpu_bo *bo = bo_va->base.bo;
>   
>   	mapping->bo_va = bo_va;
>   	list_add(&mapping->list, &bo_va->invalids);
>   	amdgpu_vm_it_insert(mapping, &vm->va);
> +	if (!bo_va->base.moved)
> +		amdgpu_vm_bo_dirty(&bo_va->base);
>   
>   	if (mapping->flags & AMDGPU_PTE_PRT)
>   		amdgpu_vm_prt_get(adev);
>   
> -	if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -	    !bo_va->base.moved) {
> -		amdgpu_vm_bo_moved(&bo_va->base);
> -	}
>   	trace_amdgpu_vm_bo_map(bo_va, mapping);
>   }
>   
> @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   			before->flags = tmp->flags;
>   			before->bo_va = tmp->bo_va;
>   			list_add(&before->list, &tmp->bo_va->invalids);
> +			if (!tmp->bo_va->base.moved)
> +				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>   		}
>   
>   		/* Remember mapping split at the end */
> @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   			after->flags = tmp->flags;
>   			after->bo_va = tmp->bo_va;
>   			list_add(&after->list, &tmp->bo_va->invalids);
> +			if (!tmp->bo_va->base.moved)
> +				amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>   		}
>   
>   		list_del(&tmp->list);
> @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   
>   	/* Insert partial mapping before the range */
>   	if (!list_empty(&before->list)) {
> -		struct amdgpu_bo *bo = before->bo_va->base.bo;
> -
>   		amdgpu_vm_it_insert(before, &vm->va);
>   		if (before->flags & AMDGPU_PTE_PRT)
>   			amdgpu_vm_prt_get(adev);
> -
> -		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -		    !before->bo_va->base.moved)
> -			amdgpu_vm_bo_moved(&before->bo_va->base);
>   	} else {
>   		kfree(before);
>   	}
>   
>   	/* Insert partial mapping after the range */
>   	if (!list_empty(&after->list)) {
> -		struct amdgpu_bo *bo = after->bo_va->base.bo;
> -
>   		amdgpu_vm_it_insert(after, &vm->va);
>   		if (after->flags & AMDGPU_PTE_PRT)
>   			amdgpu_vm_prt_get(adev);
> -
> -		if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> -		    !after->bo_va->base.moved)
> -			amdgpu_vm_bo_moved(&after->bo_va->base);
>   	} else {
>   		kfree(after);
>   	}
> @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>   	INIT_LIST_HEAD(&vm->evicted);
>   	INIT_LIST_HEAD(&vm->relocated);
>   	INIT_LIST_HEAD(&vm->moved);
> +	INIT_LIST_HEAD(&vm->dirty);
>   	INIT_LIST_HEAD(&vm->idle);
>   	INIT_LIST_HEAD(&vm->invalidated);
>   	spin_lock_init(&vm->status_lock);
> @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   {
>   	struct amdgpu_bo_va *bo_va, *tmp;
>   	u64 total_idle = 0;
> +	u64 total_dirty = 0;
>   	u64 total_relocated = 0;
>   	u64 total_moved = 0;
>   	u64 total_invalidated = 0;
>   	u64 total_done = 0;
>   	unsigned int total_idle_objs = 0;
> +	unsigned int total_dirty_objs = 0;
>   	unsigned int total_relocated_objs = 0;
>   	unsigned int total_moved_objs = 0;
>   	unsigned int total_invalidated_objs = 0;
> @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   	total_idle_objs = id;
>   	id = 0;
>   
> +	seq_puts(m, "\tDirty BOs:\n");
> +	list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
> +		if (!bo_va->base.bo)
> +			continue;
> +		total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
> +	}
> +	total_dirty_objs = id;
> +	id = 0;
> +
>   	seq_puts(m, "\tRelocated BOs:\n");
>   	list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>   		if (!bo_va->base.bo)
> @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>   
>   	seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>   		   total_idle_objs);
> +	seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
> +		   total_dirty_objs);
>   	seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>   		   total_relocated_objs);
>   	seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index d9ab97eabda9..f91d4fcf80b8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -276,6 +276,9 @@ struct amdgpu_vm {
>   	/* per VM BOs moved, but not yet updated in the PT */
>   	struct list_head	moved;
>   
> +	/* normal and per VM BOs that are not moved, but have new PT entries */
> +	struct list_head	dirty;
> +
>   	/* All BOs of this VM not currently in the state machine */
>   	struct list_head	idle;
>   

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 11751 bytes --]

On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com>
wrote:

> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
> > BO specified in the GEM ioctl; however, when a binding is split, the
> > adjacent bindings also need to be updated. Such updates currently ends up
> > getting deferred until next submission which causes stalls.
>
> Yeah, that is a necessity. The hardware simply doesn't support what you
> try to do here in all cases.
>

What can the hardware not do here? Is this just needing to wait for TLB
flushes before we can free pagetables, can we just delay that?


>
> So this approach won't work in general.
>
> Regards,
> Christian.
>
> >
> > Introduce a new state "dirty", shared between per-VM BOs and traditional
> > BOs, containing all BOs that have pending updates in `invalids`.
> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
> > in the dirty state.
> >
> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
> >   3 files changed, 63 insertions(+), 24 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > index a1b15d0d6c48..01d3a97248b0 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device
> *dev, void *data,
> >    * vital here, so they are not reported back to userspace.
> >    */
> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> > -                                 struct amdgpu_vm *vm,
> > -                                 struct amdgpu_bo_va *bo_va,
> > -                                 uint32_t operation)
> > +                                 struct amdgpu_vm *vm)
> >   {
> > +     struct amdgpu_bo_va *bo_va;
> >       int r;
> >
> >       if (!amdgpu_vm_ready(vm))
> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
> amdgpu_device *adev,
> >       if (r)
> >               goto error;
> >
> > -     if (operation == AMDGPU_VA_OP_MAP ||
> > -         operation == AMDGPU_VA_OP_REPLACE) {
> > +     spin_lock(&vm->status_lock);
> > +     while (!list_empty(&vm->dirty)) {
> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> > +                                      base.vm_status);
> > +             spin_unlock(&vm->status_lock);
> > +
> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
> >               if (r)
> >                       goto error;
> > +             spin_lock(&vm->status_lock);
> >       }
> > +     spin_unlock(&vm->status_lock);
> >
> >       r = amdgpu_vm_update_pdes(adev, vm, false);
> >
> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void
> *data,
> >               break;
> >       }
> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
> !amdgpu_vm_debug)
> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> > -                                     args->operation);
> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
> >
> >   error:
> >       drm_exec_fini(&exec);
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index dd6f72e2a1d6..01d31891cd05 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
> amdgpu_vm_bo_base *vm_bo, bool evict
> >       spin_unlock(&vm_bo->vm->status_lock);
> >   }
> >
> > +/**
> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
> > + *
> > + * @vm_bo: vm_bo which is dirty
> > + *
> > + * State for normal and per VM BOs that are not moved, but have new
> entries in
> > + * bo_va->invalids.
> > + */
> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> > +{
> > +     spin_lock(&vm_bo->vm->status_lock);
> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> > +     spin_unlock(&vm_bo->vm->status_lock);
> > +}
> > +
> >   /**
> >    * amdgpu_vm_bo_moved - vm_bo is moved
> >    *
> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
> base.eviction_status)
> >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >
> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
> > +
> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> base.vm_status)
> >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >
> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device
> *adev,
> >                       dma_resv_unlock(resv);
> >               spin_lock(&vm->status_lock);
> >       }
> > +
> > +     while (!list_empty(&vm->dirty)) {
> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> > +                                      base.vm_status);
> > +             spin_unlock(&vm->status_lock);
> > +
> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
> > +             if (r)
> > +                     return r;
> > +             spin_lock(&vm->status_lock);
> > +     }
> >       spin_unlock(&vm->status_lock);
> >
> >       return 0;
> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct
> amdgpu_device *adev,
> >                                   struct amdgpu_bo_va_mapping *mapping)
> >   {
> >       struct amdgpu_vm *vm = bo_va->base.vm;
> > -     struct amdgpu_bo *bo = bo_va->base.bo;
> >
> >       mapping->bo_va = bo_va;
> >       list_add(&mapping->list, &bo_va->invalids);
> >       amdgpu_vm_it_insert(mapping, &vm->va);
> > +     if (!bo_va->base.moved)
> > +             amdgpu_vm_bo_dirty(&bo_va->base);
> >
> >       if (mapping->flags & AMDGPU_PTE_PRT)
> >               amdgpu_vm_prt_get(adev);
> >
> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> > -         !bo_va->base.moved) {
> > -             amdgpu_vm_bo_moved(&bo_va->base);
> > -     }
> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
> >   }
> >
> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >                       before->flags = tmp->flags;
> >                       before->bo_va = tmp->bo_va;
> >                       list_add(&before->list, &tmp->bo_va->invalids);
> > +                     if (!tmp->bo_va->base.moved)
> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >               }
> >
> >               /* Remember mapping split at the end */
> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >                       after->flags = tmp->flags;
> >                       after->bo_va = tmp->bo_va;
> >                       list_add(&after->list, &tmp->bo_va->invalids);
> > +                     if (!tmp->bo_va->base.moved)
> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >               }
> >
> >               list_del(&tmp->list);
> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >
> >       /* Insert partial mapping before the range */
> >       if (!list_empty(&before->list)) {
> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
> > -
> >               amdgpu_vm_it_insert(before, &vm->va);
> >               if (before->flags & AMDGPU_PTE_PRT)
> >                       amdgpu_vm_prt_get(adev);
> > -
> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
> &&
> > -                 !before->bo_va->base.moved)
> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
> >       } else {
> >               kfree(before);
> >       }
> >
> >       /* Insert partial mapping after the range */
> >       if (!list_empty(&after->list)) {
> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
> > -
> >               amdgpu_vm_it_insert(after, &vm->va);
> >               if (after->flags & AMDGPU_PTE_PRT)
> >                       amdgpu_vm_prt_get(adev);
> > -
> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
> &&
> > -                 !after->bo_va->base.moved)
> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
> >       } else {
> >               kfree(after);
> >       }
> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev,
> struct amdgpu_vm *vm, int32_t xcp
> >       INIT_LIST_HEAD(&vm->evicted);
> >       INIT_LIST_HEAD(&vm->relocated);
> >       INIT_LIST_HEAD(&vm->moved);
> > +     INIT_LIST_HEAD(&vm->dirty);
> >       INIT_LIST_HEAD(&vm->idle);
> >       INIT_LIST_HEAD(&vm->invalidated);
> >       spin_lock_init(&vm->status_lock);
> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >   {
> >       struct amdgpu_bo_va *bo_va, *tmp;
> >       u64 total_idle = 0;
> > +     u64 total_dirty = 0;
> >       u64 total_relocated = 0;
> >       u64 total_moved = 0;
> >       u64 total_invalidated = 0;
> >       u64 total_done = 0;
> >       unsigned int total_idle_objs = 0;
> > +     unsigned int total_dirty_objs = 0;
> >       unsigned int total_relocated_objs = 0;
> >       unsigned int total_moved_objs = 0;
> >       unsigned int total_invalidated_objs = 0;
> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >       total_idle_objs = id;
> >       id = 0;
> >
> > +     seq_puts(m, "\tDirty BOs:\n");
> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
> > +             if (!bo_va->base.bo)
> > +                     continue;
> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo,
> m);
> > +     }
> > +     total_dirty_objs = id;
> > +     id = 0;
> > +
> >       seq_puts(m, "\tRelocated BOs:\n");
> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> base.vm_status) {
> >               if (!bo_va->base.bo)
> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >
> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n",
> total_idle,
> >                  total_idle_objs);
> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n",
> total_dirty,
> > +                total_dirty_objs);
> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n",
> total_relocated,
> >                  total_relocated_objs);
> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n",
> total_moved,
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > index d9ab97eabda9..f91d4fcf80b8 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
> >       /* per VM BOs moved, but not yet updated in the PT */
> >       struct list_head        moved;
> >
> > +     /* normal and per VM BOs that are not moved, but have new PT
> entries */
> > +     struct list_head        dirty;
> > +
> >       /* All BOs of this VM not currently in the state machine */
> >       struct list_head        idle;
> >
>
>

[-- Attachment #2: Type: text/html, Size: 15682 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 11751 bytes --]

On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com>
wrote:

> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
> > BO specified in the GEM ioctl; however, when a binding is split, the
> > adjacent bindings also need to be updated. Such updates currently ends up
> > getting deferred until next submission which causes stalls.
>
> Yeah, that is a necessity. The hardware simply doesn't support what you
> try to do here in all cases.
>

What can the hardware not do here? Is this just needing to wait for TLB
flushes before we can free pagetables, can we just delay that?


>
> So this approach won't work in general.
>
> Regards,
> Christian.
>
> >
> > Introduce a new state "dirty", shared between per-VM BOs and traditional
> > BOs, containing all BOs that have pending updates in `invalids`.
> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
> > in the dirty state.
> >
> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
> >   3 files changed, 63 insertions(+), 24 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > index a1b15d0d6c48..01d3a97248b0 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device
> *dev, void *data,
> >    * vital here, so they are not reported back to userspace.
> >    */
> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> > -                                 struct amdgpu_vm *vm,
> > -                                 struct amdgpu_bo_va *bo_va,
> > -                                 uint32_t operation)
> > +                                 struct amdgpu_vm *vm)
> >   {
> > +     struct amdgpu_bo_va *bo_va;
> >       int r;
> >
> >       if (!amdgpu_vm_ready(vm))
> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
> amdgpu_device *adev,
> >       if (r)
> >               goto error;
> >
> > -     if (operation == AMDGPU_VA_OP_MAP ||
> > -         operation == AMDGPU_VA_OP_REPLACE) {
> > +     spin_lock(&vm->status_lock);
> > +     while (!list_empty(&vm->dirty)) {
> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> > +                                      base.vm_status);
> > +             spin_unlock(&vm->status_lock);
> > +
> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
> >               if (r)
> >                       goto error;
> > +             spin_lock(&vm->status_lock);
> >       }
> > +     spin_unlock(&vm->status_lock);
> >
> >       r = amdgpu_vm_update_pdes(adev, vm, false);
> >
> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void
> *data,
> >               break;
> >       }
> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
> !amdgpu_vm_debug)
> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> > -                                     args->operation);
> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
> >
> >   error:
> >       drm_exec_fini(&exec);
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > index dd6f72e2a1d6..01d31891cd05 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
> amdgpu_vm_bo_base *vm_bo, bool evict
> >       spin_unlock(&vm_bo->vm->status_lock);
> >   }
> >
> > +/**
> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
> > + *
> > + * @vm_bo: vm_bo which is dirty
> > + *
> > + * State for normal and per VM BOs that are not moved, but have new
> entries in
> > + * bo_va->invalids.
> > + */
> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> > +{
> > +     spin_lock(&vm_bo->vm->status_lock);
> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> > +     spin_unlock(&vm_bo->vm->status_lock);
> > +}
> > +
> >   /**
> >    * amdgpu_vm_bo_moved - vm_bo is moved
> >    *
> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
> base.eviction_status)
> >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >
> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
> > +
> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> base.vm_status)
> >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >
> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device
> *adev,
> >                       dma_resv_unlock(resv);
> >               spin_lock(&vm->status_lock);
> >       }
> > +
> > +     while (!list_empty(&vm->dirty)) {
> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
> > +                                      base.vm_status);
> > +             spin_unlock(&vm->status_lock);
> > +
> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
> > +             if (r)
> > +                     return r;
> > +             spin_lock(&vm->status_lock);
> > +     }
> >       spin_unlock(&vm->status_lock);
> >
> >       return 0;
> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct
> amdgpu_device *adev,
> >                                   struct amdgpu_bo_va_mapping *mapping)
> >   {
> >       struct amdgpu_vm *vm = bo_va->base.vm;
> > -     struct amdgpu_bo *bo = bo_va->base.bo;
> >
> >       mapping->bo_va = bo_va;
> >       list_add(&mapping->list, &bo_va->invalids);
> >       amdgpu_vm_it_insert(mapping, &vm->va);
> > +     if (!bo_va->base.moved)
> > +             amdgpu_vm_bo_dirty(&bo_va->base);
> >
> >       if (mapping->flags & AMDGPU_PTE_PRT)
> >               amdgpu_vm_prt_get(adev);
> >
> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> > -         !bo_va->base.moved) {
> > -             amdgpu_vm_bo_moved(&bo_va->base);
> > -     }
> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
> >   }
> >
> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >                       before->flags = tmp->flags;
> >                       before->bo_va = tmp->bo_va;
> >                       list_add(&before->list, &tmp->bo_va->invalids);
> > +                     if (!tmp->bo_va->base.moved)
> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >               }
> >
> >               /* Remember mapping split at the end */
> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >                       after->flags = tmp->flags;
> >                       after->bo_va = tmp->bo_va;
> >                       list_add(&after->list, &tmp->bo_va->invalids);
> > +                     if (!tmp->bo_va->base.moved)
> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >               }
> >
> >               list_del(&tmp->list);
> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
> amdgpu_device *adev,
> >
> >       /* Insert partial mapping before the range */
> >       if (!list_empty(&before->list)) {
> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
> > -
> >               amdgpu_vm_it_insert(before, &vm->va);
> >               if (before->flags & AMDGPU_PTE_PRT)
> >                       amdgpu_vm_prt_get(adev);
> > -
> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
> &&
> > -                 !before->bo_va->base.moved)
> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
> >       } else {
> >               kfree(before);
> >       }
> >
> >       /* Insert partial mapping after the range */
> >       if (!list_empty(&after->list)) {
> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
> > -
> >               amdgpu_vm_it_insert(after, &vm->va);
> >               if (after->flags & AMDGPU_PTE_PRT)
> >                       amdgpu_vm_prt_get(adev);
> > -
> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
> &&
> > -                 !after->bo_va->base.moved)
> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
> >       } else {
> >               kfree(after);
> >       }
> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev,
> struct amdgpu_vm *vm, int32_t xcp
> >       INIT_LIST_HEAD(&vm->evicted);
> >       INIT_LIST_HEAD(&vm->relocated);
> >       INIT_LIST_HEAD(&vm->moved);
> > +     INIT_LIST_HEAD(&vm->dirty);
> >       INIT_LIST_HEAD(&vm->idle);
> >       INIT_LIST_HEAD(&vm->invalidated);
> >       spin_lock_init(&vm->status_lock);
> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >   {
> >       struct amdgpu_bo_va *bo_va, *tmp;
> >       u64 total_idle = 0;
> > +     u64 total_dirty = 0;
> >       u64 total_relocated = 0;
> >       u64 total_moved = 0;
> >       u64 total_invalidated = 0;
> >       u64 total_done = 0;
> >       unsigned int total_idle_objs = 0;
> > +     unsigned int total_dirty_objs = 0;
> >       unsigned int total_relocated_objs = 0;
> >       unsigned int total_moved_objs = 0;
> >       unsigned int total_invalidated_objs = 0;
> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >       total_idle_objs = id;
> >       id = 0;
> >
> > +     seq_puts(m, "\tDirty BOs:\n");
> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
> > +             if (!bo_va->base.bo)
> > +                     continue;
> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo,
> m);
> > +     }
> > +     total_dirty_objs = id;
> > +     id = 0;
> > +
> >       seq_puts(m, "\tRelocated BOs:\n");
> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> base.vm_status) {
> >               if (!bo_va->base.bo)
> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
> *vm, struct seq_file *m)
> >
> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n",
> total_idle,
> >                  total_idle_objs);
> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n",
> total_dirty,
> > +                total_dirty_objs);
> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n",
> total_relocated,
> >                  total_relocated_objs);
> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n",
> total_moved,
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > index d9ab97eabda9..f91d4fcf80b8 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
> >       /* per VM BOs moved, but not yet updated in the PT */
> >       struct list_head        moved;
> >
> > +     /* normal and per VM BOs that are not moved, but have new PT
> entries */
> > +     struct list_head        dirty;
> > +
> >       /* All BOs of this VM not currently in the state machine */
> >       struct list_head        idle;
> >
>
>

[-- Attachment #2: Type: text/html, Size: 15682 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 14508 bytes --]

Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>
>
> On Tue, Oct 31, 2023 at 2:57 PM Christian König 
> <christian.koenig@amd.com> wrote:
>
>     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>     > The current amdgpu_gem_va_update_vm only tries to perform
>     updates for the
>     > BO specified in the GEM ioctl; however, when a binding is split, the
>     > adjacent bindings also need to be updated. Such updates
>     currently ends up
>     > getting deferred until next submission which causes stalls.
>
>     Yeah, that is a necessity. The hardware simply doesn't support
>     what you
>     try to do here in all cases.
>
>
> What can the hardware not do here? Is this just needing to wait for 
> TLB flushes before we can free pagetables, can we just delay that?

On some hardware generations (especially Navi1x, but also everything 
older than Polaris) you can't invalidate the TLB while it is in use.

For Polaris and older it just means that you don't have a guarantee that 
the shader can't access the memory any more. So delaying the free 
operation helps here.

But for Navi1x it's a workaround for a hardware bug. If you try to 
invalidate the TLB while it is in use you can potentially triggering 
memory accesses to random addresses.

That's why we still delay TLB invalidation's to the next CS and use a 
new VMID for each submission instead of invalidating the old one.

I'm currently working on changing that for Navi2x and newer (maybe Vega 
as well), but this is something you can really only do on some hw 
generations after validating that it works.

Regards,
Christian.

>
>
>     So this approach won't work in general.
>
>     Regards,
>     Christian.
>
>     >
>     > Introduce a new state "dirty", shared between per-VM BOs and
>     traditional
>     > BOs, containing all BOs that have pending updates in `invalids`.
>     > amdgpu_gem_va_update_vm will now simply flush any pending
>     updates for BOs
>     > in the dirty state.
>     >
>     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>     > ---
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>     ++++++++++++++++++-------
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>     >   3 files changed, 63 insertions(+), 24 deletions(-)
>     >
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > index a1b15d0d6c48..01d3a97248b0 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>     drm_device *dev, void *data,
>     >    * vital here, so they are not reported back to userspace.
>     >    */
>     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>     > -                                 struct amdgpu_vm *vm,
>     > -                                 struct amdgpu_bo_va *bo_va,
>     > -                                 uint32_t operation)
>     > +                                 struct amdgpu_vm *vm)
>     >   {
>     > +     struct amdgpu_bo_va *bo_va;
>     >       int r;
>     >
>     >       if (!amdgpu_vm_ready(vm))
>     > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>     amdgpu_device *adev,
>     >       if (r)
>     >               goto error;
>     >
>     > -     if (operation == AMDGPU_VA_OP_MAP ||
>     > -         operation == AMDGPU_VA_OP_REPLACE) {
>     > +     spin_lock(&vm->status_lock);
>     > +     while (!list_empty(&vm->dirty)) {
>     > +             bo_va = list_first_entry(&vm->dirty, struct
>     amdgpu_bo_va,
>     > +                                      base.vm_status);
>     > +             spin_unlock(&vm->status_lock);
>     > +
>     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>     >               if (r)
>     >                       goto error;
>     > +             spin_lock(&vm->status_lock);
>     >       }
>     > +     spin_unlock(&vm->status_lock);
>     >
>     >       r = amdgpu_vm_update_pdes(adev, vm, false);
>     >
>     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>     *dev, void *data,
>     >               break;
>     >       }
>     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>     !amdgpu_vm_debug)
>     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>     > -  args->operation);
>     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>     >
>     >   error:
>     >       drm_exec_fini(&exec);
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > index dd6f72e2a1d6..01d31891cd05 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>     amdgpu_vm_bo_base *vm_bo, bool evict
>     >       spin_unlock(&vm_bo->vm->status_lock);
>     >   }
>     >
>     > +/**
>     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>     > + *
>     > + * @vm_bo: vm_bo which is dirty
>     > + *
>     > + * State for normal and per VM BOs that are not moved, but have
>     new entries in
>     > + * bo_va->invalids.
>     > + */
>     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>     > +{
>     > +     spin_lock(&vm_bo->vm->status_lock);
>     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>     > +     spin_unlock(&vm_bo->vm->status_lock);
>     > +}
>     > +
>     >   /**
>     >    * amdgpu_vm_bo_moved - vm_bo is moved
>     >    *
>     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
>     *vm,
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>     base.eviction_status)
>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>     >
>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>     base.vm_status)
>     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>     > +
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>     base.vm_status)
>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>     >
>     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>     amdgpu_device *adev,
>     >                       dma_resv_unlock(resv);
>     >               spin_lock(&vm->status_lock);
>     >       }
>     > +
>     > +     while (!list_empty(&vm->dirty)) {
>     > +             bo_va = list_first_entry(&vm->dirty, struct
>     amdgpu_bo_va,
>     > +                                      base.vm_status);
>     > +             spin_unlock(&vm->status_lock);
>     > +
>     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>     > +             if (r)
>     > +                     return r;
>     > +             spin_lock(&vm->status_lock);
>     > +     }
>     >       spin_unlock(&vm->status_lock);
>     >
>     >       return 0;
>     > @@ -1476,19 +1505,16 @@ static void
>     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>     >                                   struct amdgpu_bo_va_mapping
>     *mapping)
>     >   {
>     >       struct amdgpu_vm *vm = bo_va->base.vm;
>     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
>     >
>     >       mapping->bo_va = bo_va;
>     >       list_add(&mapping->list, &bo_va->invalids);
>     >       amdgpu_vm_it_insert(mapping, &vm->va);
>     > +     if (!bo_va->base.moved)
>     > +             amdgpu_vm_bo_dirty(&bo_va->base);
>     >
>     >       if (mapping->flags & AMDGPU_PTE_PRT)
>     >               amdgpu_vm_prt_get(adev);
>     >
>     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>     > -         !bo_va->base.moved) {
>     > -             amdgpu_vm_bo_moved(&bo_va->base);
>     > -     }
>     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>     >   }
>     >
>     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >                       before->flags = tmp->flags;
>     >                       before->bo_va = tmp->bo_va;
>     >                       list_add(&before->list,
>     &tmp->bo_va->invalids);
>     > +                     if (!tmp->bo_va->base.moved)
>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>     >               }
>     >
>     >               /* Remember mapping split at the end */
>     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >                       after->flags = tmp->flags;
>     >                       after->bo_va = tmp->bo_va;
>     >                       list_add(&after->list, &tmp->bo_va->invalids);
>     > +                     if (!tmp->bo_va->base.moved)
>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>     >               }
>     >
>     >               list_del(&tmp->list);
>     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >
>     >       /* Insert partial mapping before the range */
>     >       if (!list_empty(&before->list)) {
>     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>     <http://base.bo>;
>     > -
>     >               amdgpu_vm_it_insert(before, &vm->va);
>     >               if (before->flags & AMDGPU_PTE_PRT)
>     >                       amdgpu_vm_prt_get(adev);
>     > -
>     > -             if (bo && bo->tbo.base.resv ==
>     vm->root.bo->tbo.base.resv &&
>     > -                 !before->bo_va->base.moved)
>     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>     >       } else {
>     >               kfree(before);
>     >       }
>     >
>     >       /* Insert partial mapping after the range */
>     >       if (!list_empty(&after->list)) {
>     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>     <http://base.bo>;
>     > -
>     >               amdgpu_vm_it_insert(after, &vm->va);
>     >               if (after->flags & AMDGPU_PTE_PRT)
>     >                       amdgpu_vm_prt_get(adev);
>     > -
>     > -             if (bo && bo->tbo.base.resv ==
>     vm->root.bo->tbo.base.resv &&
>     > -                 !after->bo_va->base.moved)
>     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>     >       } else {
>     >               kfree(after);
>     >       }
>     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>     *adev, struct amdgpu_vm *vm, int32_t xcp
>     >       INIT_LIST_HEAD(&vm->evicted);
>     >       INIT_LIST_HEAD(&vm->relocated);
>     >       INIT_LIST_HEAD(&vm->moved);
>     > +     INIT_LIST_HEAD(&vm->dirty);
>     >       INIT_LIST_HEAD(&vm->idle);
>     >       INIT_LIST_HEAD(&vm->invalidated);
>     >       spin_lock_init(&vm->status_lock);
>     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >   {
>     >       struct amdgpu_bo_va *bo_va, *tmp;
>     >       u64 total_idle = 0;
>     > +     u64 total_dirty = 0;
>     >       u64 total_relocated = 0;
>     >       u64 total_moved = 0;
>     >       u64 total_invalidated = 0;
>     >       u64 total_done = 0;
>     >       unsigned int total_idle_objs = 0;
>     > +     unsigned int total_dirty_objs = 0;
>     >       unsigned int total_relocated_objs = 0;
>     >       unsigned int total_moved_objs = 0;
>     >       unsigned int total_invalidated_objs = 0;
>     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >       total_idle_objs = id;
>     >       id = 0;
>     >
>     > +     seq_puts(m, "\tDirty BOs:\n");
>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>     base.vm_status) {
>     > +             if (!bo_va->base.bo <http://base.bo>)
>     > +                     continue;
>     > +             total_dirty += amdgpu_bo_print_info(id++,
>     bo_va->base.bo <http://base.bo>, m);
>     > +     }
>     > +     total_dirty_objs = id;
>     > +     id = 0;
>     > +
>     >       seq_puts(m, "\tRelocated BOs:\n");
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>     base.vm_status) {
>     >               if (!bo_va->base.bo <http://base.bo>)
>     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >
>     >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
>     total_idle,
>     >                  total_idle_objs);
>     > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
>     total_dirty,
>     > +                total_dirty_objs);
>     >       seq_printf(m, "\tTotal relocated size:
>      %12lld\tobjs:\t%d\n", total_relocated,
>     >                  total_relocated_objs);
>     >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
>     total_moved,
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > index d9ab97eabda9..f91d4fcf80b8 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>     >       /* per VM BOs moved, but not yet updated in the PT */
>     >       struct list_head        moved;
>     >
>     > +     /* normal and per VM BOs that are not moved, but have new
>     PT entries */
>     > +     struct list_head        dirty;
>     > +
>     >       /* All BOs of this VM not currently in the state machine */
>     >       struct list_head        idle;
>     >
>

[-- Attachment #2: Type: text/html, Size: 25838 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 14508 bytes --]

Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>
>
> On Tue, Oct 31, 2023 at 2:57 PM Christian König 
> <christian.koenig@amd.com> wrote:
>
>     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>     > The current amdgpu_gem_va_update_vm only tries to perform
>     updates for the
>     > BO specified in the GEM ioctl; however, when a binding is split, the
>     > adjacent bindings also need to be updated. Such updates
>     currently ends up
>     > getting deferred until next submission which causes stalls.
>
>     Yeah, that is a necessity. The hardware simply doesn't support
>     what you
>     try to do here in all cases.
>
>
> What can the hardware not do here? Is this just needing to wait for 
> TLB flushes before we can free pagetables, can we just delay that?

On some hardware generations (especially Navi1x, but also everything 
older than Polaris) you can't invalidate the TLB while it is in use.

For Polaris and older it just means that you don't have a guarantee that 
the shader can't access the memory any more. So delaying the free 
operation helps here.

But for Navi1x it's a workaround for a hardware bug. If you try to 
invalidate the TLB while it is in use you can potentially triggering 
memory accesses to random addresses.

That's why we still delay TLB invalidation's to the next CS and use a 
new VMID for each submission instead of invalidating the old one.

I'm currently working on changing that for Navi2x and newer (maybe Vega 
as well), but this is something you can really only do on some hw 
generations after validating that it works.

Regards,
Christian.

>
>
>     So this approach won't work in general.
>
>     Regards,
>     Christian.
>
>     >
>     > Introduce a new state "dirty", shared between per-VM BOs and
>     traditional
>     > BOs, containing all BOs that have pending updates in `invalids`.
>     > amdgpu_gem_va_update_vm will now simply flush any pending
>     updates for BOs
>     > in the dirty state.
>     >
>     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>     > ---
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>     ++++++++++++++++++-------
>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>     >   3 files changed, 63 insertions(+), 24 deletions(-)
>     >
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > index a1b15d0d6c48..01d3a97248b0 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>     drm_device *dev, void *data,
>     >    * vital here, so they are not reported back to userspace.
>     >    */
>     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>     > -                                 struct amdgpu_vm *vm,
>     > -                                 struct amdgpu_bo_va *bo_va,
>     > -                                 uint32_t operation)
>     > +                                 struct amdgpu_vm *vm)
>     >   {
>     > +     struct amdgpu_bo_va *bo_va;
>     >       int r;
>     >
>     >       if (!amdgpu_vm_ready(vm))
>     > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>     amdgpu_device *adev,
>     >       if (r)
>     >               goto error;
>     >
>     > -     if (operation == AMDGPU_VA_OP_MAP ||
>     > -         operation == AMDGPU_VA_OP_REPLACE) {
>     > +     spin_lock(&vm->status_lock);
>     > +     while (!list_empty(&vm->dirty)) {
>     > +             bo_va = list_first_entry(&vm->dirty, struct
>     amdgpu_bo_va,
>     > +                                      base.vm_status);
>     > +             spin_unlock(&vm->status_lock);
>     > +
>     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>     >               if (r)
>     >                       goto error;
>     > +             spin_lock(&vm->status_lock);
>     >       }
>     > +     spin_unlock(&vm->status_lock);
>     >
>     >       r = amdgpu_vm_update_pdes(adev, vm, false);
>     >
>     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>     *dev, void *data,
>     >               break;
>     >       }
>     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>     !amdgpu_vm_debug)
>     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>     > -  args->operation);
>     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>     >
>     >   error:
>     >       drm_exec_fini(&exec);
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > index dd6f72e2a1d6..01d31891cd05 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>     > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>     amdgpu_vm_bo_base *vm_bo, bool evict
>     >       spin_unlock(&vm_bo->vm->status_lock);
>     >   }
>     >
>     > +/**
>     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>     > + *
>     > + * @vm_bo: vm_bo which is dirty
>     > + *
>     > + * State for normal and per VM BOs that are not moved, but have
>     new entries in
>     > + * bo_va->invalids.
>     > + */
>     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>     > +{
>     > +     spin_lock(&vm_bo->vm->status_lock);
>     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>     > +     spin_unlock(&vm_bo->vm->status_lock);
>     > +}
>     > +
>     >   /**
>     >    * amdgpu_vm_bo_moved - vm_bo is moved
>     >    *
>     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
>     *vm,
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>     base.eviction_status)
>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>     >
>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>     base.vm_status)
>     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>     > +
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>     base.vm_status)
>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>     >
>     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>     amdgpu_device *adev,
>     >                       dma_resv_unlock(resv);
>     >               spin_lock(&vm->status_lock);
>     >       }
>     > +
>     > +     while (!list_empty(&vm->dirty)) {
>     > +             bo_va = list_first_entry(&vm->dirty, struct
>     amdgpu_bo_va,
>     > +                                      base.vm_status);
>     > +             spin_unlock(&vm->status_lock);
>     > +
>     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>     > +             if (r)
>     > +                     return r;
>     > +             spin_lock(&vm->status_lock);
>     > +     }
>     >       spin_unlock(&vm->status_lock);
>     >
>     >       return 0;
>     > @@ -1476,19 +1505,16 @@ static void
>     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>     >                                   struct amdgpu_bo_va_mapping
>     *mapping)
>     >   {
>     >       struct amdgpu_vm *vm = bo_va->base.vm;
>     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
>     >
>     >       mapping->bo_va = bo_va;
>     >       list_add(&mapping->list, &bo_va->invalids);
>     >       amdgpu_vm_it_insert(mapping, &vm->va);
>     > +     if (!bo_va->base.moved)
>     > +             amdgpu_vm_bo_dirty(&bo_va->base);
>     >
>     >       if (mapping->flags & AMDGPU_PTE_PRT)
>     >               amdgpu_vm_prt_get(adev);
>     >
>     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>     > -         !bo_va->base.moved) {
>     > -             amdgpu_vm_bo_moved(&bo_va->base);
>     > -     }
>     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>     >   }
>     >
>     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >                       before->flags = tmp->flags;
>     >                       before->bo_va = tmp->bo_va;
>     >                       list_add(&before->list,
>     &tmp->bo_va->invalids);
>     > +                     if (!tmp->bo_va->base.moved)
>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>     >               }
>     >
>     >               /* Remember mapping split at the end */
>     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >                       after->flags = tmp->flags;
>     >                       after->bo_va = tmp->bo_va;
>     >                       list_add(&after->list, &tmp->bo_va->invalids);
>     > +                     if (!tmp->bo_va->base.moved)
>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>     >               }
>     >
>     >               list_del(&tmp->list);
>     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>     amdgpu_device *adev,
>     >
>     >       /* Insert partial mapping before the range */
>     >       if (!list_empty(&before->list)) {
>     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>     <http://base.bo>;
>     > -
>     >               amdgpu_vm_it_insert(before, &vm->va);
>     >               if (before->flags & AMDGPU_PTE_PRT)
>     >                       amdgpu_vm_prt_get(adev);
>     > -
>     > -             if (bo && bo->tbo.base.resv ==
>     vm->root.bo->tbo.base.resv &&
>     > -                 !before->bo_va->base.moved)
>     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>     >       } else {
>     >               kfree(before);
>     >       }
>     >
>     >       /* Insert partial mapping after the range */
>     >       if (!list_empty(&after->list)) {
>     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>     <http://base.bo>;
>     > -
>     >               amdgpu_vm_it_insert(after, &vm->va);
>     >               if (after->flags & AMDGPU_PTE_PRT)
>     >                       amdgpu_vm_prt_get(adev);
>     > -
>     > -             if (bo && bo->tbo.base.resv ==
>     vm->root.bo->tbo.base.resv &&
>     > -                 !after->bo_va->base.moved)
>     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>     >       } else {
>     >               kfree(after);
>     >       }
>     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>     *adev, struct amdgpu_vm *vm, int32_t xcp
>     >       INIT_LIST_HEAD(&vm->evicted);
>     >       INIT_LIST_HEAD(&vm->relocated);
>     >       INIT_LIST_HEAD(&vm->moved);
>     > +     INIT_LIST_HEAD(&vm->dirty);
>     >       INIT_LIST_HEAD(&vm->idle);
>     >       INIT_LIST_HEAD(&vm->invalidated);
>     >       spin_lock_init(&vm->status_lock);
>     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >   {
>     >       struct amdgpu_bo_va *bo_va, *tmp;
>     >       u64 total_idle = 0;
>     > +     u64 total_dirty = 0;
>     >       u64 total_relocated = 0;
>     >       u64 total_moved = 0;
>     >       u64 total_invalidated = 0;
>     >       u64 total_done = 0;
>     >       unsigned int total_idle_objs = 0;
>     > +     unsigned int total_dirty_objs = 0;
>     >       unsigned int total_relocated_objs = 0;
>     >       unsigned int total_moved_objs = 0;
>     >       unsigned int total_invalidated_objs = 0;
>     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >       total_idle_objs = id;
>     >       id = 0;
>     >
>     > +     seq_puts(m, "\tDirty BOs:\n");
>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>     base.vm_status) {
>     > +             if (!bo_va->base.bo <http://base.bo>)
>     > +                     continue;
>     > +             total_dirty += amdgpu_bo_print_info(id++,
>     bo_va->base.bo <http://base.bo>, m);
>     > +     }
>     > +     total_dirty_objs = id;
>     > +     id = 0;
>     > +
>     >       seq_puts(m, "\tRelocated BOs:\n");
>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>     base.vm_status) {
>     >               if (!bo_va->base.bo <http://base.bo>)
>     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>     amdgpu_vm *vm, struct seq_file *m)
>     >
>     >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
>     total_idle,
>     >                  total_idle_objs);
>     > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
>     total_dirty,
>     > +                total_dirty_objs);
>     >       seq_printf(m, "\tTotal relocated size:
>      %12lld\tobjs:\t%d\n", total_relocated,
>     >                  total_relocated_objs);
>     >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
>     total_moved,
>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > index d9ab97eabda9..f91d4fcf80b8 100644
>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>     >       /* per VM BOs moved, but not yet updated in the PT */
>     >       struct list_head        moved;
>     >
>     > +     /* normal and per VM BOs that are not moved, but have new
>     PT entries */
>     > +     struct list_head        dirty;
>     > +
>     >       /* All BOs of this VM not currently in the state machine */
>     >       struct list_head        idle;
>     >
>

[-- Attachment #2: Type: text/html, Size: 25838 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 13604 bytes --]

On Tue, Oct 31, 2023 at 3:08 PM Christian König <christian.koenig@amd.com>
wrote:

> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>
>
>
> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com>
> wrote:
>
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> > The current amdgpu_gem_va_update_vm only tries to perform updates for
>> the
>> > BO specified in the GEM ioctl; however, when a binding is split, the
>> > adjacent bindings also need to be updated. Such updates currently ends
>> up
>> > getting deferred until next submission which causes stalls.
>>
>> Yeah, that is a necessity. The hardware simply doesn't support what you
>> try to do here in all cases.
>>
>
> What can the hardware not do here? Is this just needing to wait for TLB
> flushes before we can free pagetables, can we just delay that?
>
>
> On some hardware generations (especially Navi1x, but also everything older
> than Polaris) you can't invalidate the TLB while it is in use.
>
> For Polaris and older it just means that you don't have a guarantee that
> the shader can't access the memory any more. So delaying the free operation
> helps here.
>
> But for Navi1x it's a workaround for a hardware bug. If you try to
> invalidate the TLB while it is in use you can potentially triggering memory
> accesses to random addresses.
>
> That's why we still delay TLB invalidation's to the next CS and use a new
> VMID for each submission instead of invalidating the old one.
>
> I'm currently working on changing that for Navi2x and newer (maybe Vega as
> well), but this is something you can really only do on some hw generations
> after validating that it works.
>

I think as long as we make sure all significant work gets done
asynchronously, doing the TLB flushing on the next submit (/submissions,
one per queue?) is fine for our purposes.

(As an aside after thinking some more I *think* we also need some work to
make these maps/unmaps (VALID->PRT and PRT->VALID) atomic, as I think it is
valid Vulkan to make these race. As such I'm speculating we'd need a bit
more reworking there too, not just a late free of the lower level
pagetables)

- Bas

>
> Regards,
> Christian.
>
>
>
>>
>> So this approach won't work in general.
>>
>> Regards,
>> Christian.
>>
>> >
>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>> > BOs, containing all BOs that have pending updates in `invalids`.
>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for
>> BOs
>> > in the dirty state.
>> >
>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> > ---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>> >
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > index a1b15d0d6c48..01d3a97248b0 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device
>> *dev, void *data,
>> >    * vital here, so they are not reported back to userspace.
>> >    */
>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> > -                                 struct amdgpu_vm *vm,
>> > -                                 struct amdgpu_bo_va *bo_va,
>> > -                                 uint32_t operation)
>> > +                                 struct amdgpu_vm *vm)
>> >   {
>> > +     struct amdgpu_bo_va *bo_va;
>> >       int r;
>> >
>> >       if (!amdgpu_vm_ready(vm))
>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>> amdgpu_device *adev,
>> >       if (r)
>> >               goto error;
>> >
>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>> > +     spin_lock(&vm->status_lock);
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>> >               if (r)
>> >                       goto error;
>> > +             spin_lock(&vm->status_lock);
>> >       }
>> > +     spin_unlock(&vm->status_lock);
>> >
>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>> >
>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev,
>> void *data,
>> >               break;
>> >       }
>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>> !amdgpu_vm_debug)
>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>> > -                                     args->operation);
>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>> >
>> >   error:
>> >       drm_exec_fini(&exec);
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > index dd6f72e2a1d6..01d31891cd05 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>> amdgpu_vm_bo_base *vm_bo, bool evict
>> >       spin_unlock(&vm_bo->vm->status_lock);
>> >   }
>> >
>> > +/**
>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>> > + *
>> > + * @vm_bo: vm_bo which is dirty
>> > + *
>> > + * State for normal and per VM BOs that are not moved, but have new
>> entries in
>> > + * bo_va->invalids.
>> > + */
>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>> > +{
>> > +     spin_lock(&vm_bo->vm->status_lock);
>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>> > +     spin_unlock(&vm_bo->vm->status_lock);
>> > +}
>> > +
>> >   /**
>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>> >    *
>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>> base.eviction_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>> > +
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>> base.vm_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >
>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device
>> *adev,
>> >                       dma_resv_unlock(resv);
>> >               spin_lock(&vm->status_lock);
>> >       }
>> > +
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>> > +             if (r)
>> > +                     return r;
>> > +             spin_lock(&vm->status_lock);
>> > +     }
>> >       spin_unlock(&vm->status_lock);
>> >
>> >       return 0;
>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct
>> amdgpu_device *adev,
>> >                                   struct amdgpu_bo_va_mapping *mapping)
>> >   {
>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>> > -     struct amdgpu_bo *bo = bo_va->base.bo;
>> >
>> >       mapping->bo_va = bo_va;
>> >       list_add(&mapping->list, &bo_va->invalids);
>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>> > +     if (!bo_va->base.moved)
>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>> >
>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>> >               amdgpu_vm_prt_get(adev);
>> >
>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -         !bo_va->base.moved) {
>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>> > -     }
>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>> >   }
>> >
>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >                       before->flags = tmp->flags;
>> >                       before->bo_va = tmp->bo_va;
>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >
>> >               /* Remember mapping split at the end */
>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >                       after->flags = tmp->flags;
>> >                       after->bo_va = tmp->bo_va;
>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >
>> >               list_del(&tmp->list);
>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >
>> >       /* Insert partial mapping before the range */
>> >       if (!list_empty(&before->list)) {
>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(before, &vm->va);
>> >               if (before->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
>> &&
>> > -                 !before->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>> >       } else {
>> >               kfree(before);
>> >       }
>> >
>> >       /* Insert partial mapping after the range */
>> >       if (!list_empty(&after->list)) {
>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(after, &vm->va);
>> >               if (after->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
>> &&
>> > -                 !after->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>> >       } else {
>> >               kfree(after);
>> >       }
>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev,
>> struct amdgpu_vm *vm, int32_t xcp
>> >       INIT_LIST_HEAD(&vm->evicted);
>> >       INIT_LIST_HEAD(&vm->relocated);
>> >       INIT_LIST_HEAD(&vm->moved);
>> > +     INIT_LIST_HEAD(&vm->dirty);
>> >       INIT_LIST_HEAD(&vm->idle);
>> >       INIT_LIST_HEAD(&vm->invalidated);
>> >       spin_lock_init(&vm->status_lock);
>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >   {
>> >       struct amdgpu_bo_va *bo_va, *tmp;
>> >       u64 total_idle = 0;
>> > +     u64 total_dirty = 0;
>> >       u64 total_relocated = 0;
>> >       u64 total_moved = 0;
>> >       u64 total_invalidated = 0;
>> >       u64 total_done = 0;
>> >       unsigned int total_idle_objs = 0;
>> > +     unsigned int total_dirty_objs = 0;
>> >       unsigned int total_relocated_objs = 0;
>> >       unsigned int total_moved_objs = 0;
>> >       unsigned int total_invalidated_objs = 0;
>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >       total_idle_objs = id;
>> >       id = 0;
>> >
>> > +     seq_puts(m, "\tDirty BOs:\n");
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>> > +             if (!bo_va->base.bo)
>> > +                     continue;
>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo,
>> m);
>> > +     }
>> > +     total_dirty_objs = id;
>> > +     id = 0;
>> > +
>> >       seq_puts(m, "\tRelocated BOs:\n");
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>> base.vm_status) {
>> >               if (!bo_va->base.bo)
>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >
>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n",
>> total_idle,
>> >                  total_idle_objs);
>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n",
>> total_dirty,
>> > +                total_dirty_objs);
>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n",
>> total_relocated,
>> >                  total_relocated_objs);
>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n",
>> total_moved,
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > index d9ab97eabda9..f91d4fcf80b8 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>> >       /* per VM BOs moved, but not yet updated in the PT */
>> >       struct list_head        moved;
>> >
>> > +     /* normal and per VM BOs that are not moved, but have new PT
>> entries */
>> > +     struct list_head        dirty;
>> > +
>> >       /* All BOs of this VM not currently in the state machine */
>> >       struct list_head        idle;
>> >
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 22508 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 13604 bytes --]

On Tue, Oct 31, 2023 at 3:08 PM Christian König <christian.koenig@amd.com>
wrote:

> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>
>
>
> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com>
> wrote:
>
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> > The current amdgpu_gem_va_update_vm only tries to perform updates for
>> the
>> > BO specified in the GEM ioctl; however, when a binding is split, the
>> > adjacent bindings also need to be updated. Such updates currently ends
>> up
>> > getting deferred until next submission which causes stalls.
>>
>> Yeah, that is a necessity. The hardware simply doesn't support what you
>> try to do here in all cases.
>>
>
> What can the hardware not do here? Is this just needing to wait for TLB
> flushes before we can free pagetables, can we just delay that?
>
>
> On some hardware generations (especially Navi1x, but also everything older
> than Polaris) you can't invalidate the TLB while it is in use.
>
> For Polaris and older it just means that you don't have a guarantee that
> the shader can't access the memory any more. So delaying the free operation
> helps here.
>
> But for Navi1x it's a workaround for a hardware bug. If you try to
> invalidate the TLB while it is in use you can potentially triggering memory
> accesses to random addresses.
>
> That's why we still delay TLB invalidation's to the next CS and use a new
> VMID for each submission instead of invalidating the old one.
>
> I'm currently working on changing that for Navi2x and newer (maybe Vega as
> well), but this is something you can really only do on some hw generations
> after validating that it works.
>

I think as long as we make sure all significant work gets done
asynchronously, doing the TLB flushing on the next submit (/submissions,
one per queue?) is fine for our purposes.

(As an aside after thinking some more I *think* we also need some work to
make these maps/unmaps (VALID->PRT and PRT->VALID) atomic, as I think it is
valid Vulkan to make these race. As such I'm speculating we'd need a bit
more reworking there too, not just a late free of the lower level
pagetables)

- Bas

>
> Regards,
> Christian.
>
>
>
>>
>> So this approach won't work in general.
>>
>> Regards,
>> Christian.
>>
>> >
>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>> > BOs, containing all BOs that have pending updates in `invalids`.
>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for
>> BOs
>> > in the dirty state.
>> >
>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> > ---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>> >
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > index a1b15d0d6c48..01d3a97248b0 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device
>> *dev, void *data,
>> >    * vital here, so they are not reported back to userspace.
>> >    */
>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> > -                                 struct amdgpu_vm *vm,
>> > -                                 struct amdgpu_bo_va *bo_va,
>> > -                                 uint32_t operation)
>> > +                                 struct amdgpu_vm *vm)
>> >   {
>> > +     struct amdgpu_bo_va *bo_va;
>> >       int r;
>> >
>> >       if (!amdgpu_vm_ready(vm))
>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>> amdgpu_device *adev,
>> >       if (r)
>> >               goto error;
>> >
>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>> > +     spin_lock(&vm->status_lock);
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>> >               if (r)
>> >                       goto error;
>> > +             spin_lock(&vm->status_lock);
>> >       }
>> > +     spin_unlock(&vm->status_lock);
>> >
>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>> >
>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev,
>> void *data,
>> >               break;
>> >       }
>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>> !amdgpu_vm_debug)
>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>> > -                                     args->operation);
>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>> >
>> >   error:
>> >       drm_exec_fini(&exec);
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > index dd6f72e2a1d6..01d31891cd05 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>> amdgpu_vm_bo_base *vm_bo, bool evict
>> >       spin_unlock(&vm_bo->vm->status_lock);
>> >   }
>> >
>> > +/**
>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>> > + *
>> > + * @vm_bo: vm_bo which is dirty
>> > + *
>> > + * State for normal and per VM BOs that are not moved, but have new
>> entries in
>> > + * bo_va->invalids.
>> > + */
>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>> > +{
>> > +     spin_lock(&vm_bo->vm->status_lock);
>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>> > +     spin_unlock(&vm_bo->vm->status_lock);
>> > +}
>> > +
>> >   /**
>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>> >    *
>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>> base.eviction_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>> > +
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>> base.vm_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >
>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device
>> *adev,
>> >                       dma_resv_unlock(resv);
>> >               spin_lock(&vm->status_lock);
>> >       }
>> > +
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>> > +             if (r)
>> > +                     return r;
>> > +             spin_lock(&vm->status_lock);
>> > +     }
>> >       spin_unlock(&vm->status_lock);
>> >
>> >       return 0;
>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct
>> amdgpu_device *adev,
>> >                                   struct amdgpu_bo_va_mapping *mapping)
>> >   {
>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>> > -     struct amdgpu_bo *bo = bo_va->base.bo;
>> >
>> >       mapping->bo_va = bo_va;
>> >       list_add(&mapping->list, &bo_va->invalids);
>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>> > +     if (!bo_va->base.moved)
>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>> >
>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>> >               amdgpu_vm_prt_get(adev);
>> >
>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -         !bo_va->base.moved) {
>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>> > -     }
>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>> >   }
>> >
>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >                       before->flags = tmp->flags;
>> >                       before->bo_va = tmp->bo_va;
>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >
>> >               /* Remember mapping split at the end */
>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >                       after->flags = tmp->flags;
>> >                       after->bo_va = tmp->bo_va;
>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >
>> >               list_del(&tmp->list);
>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>> amdgpu_device *adev,
>> >
>> >       /* Insert partial mapping before the range */
>> >       if (!list_empty(&before->list)) {
>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(before, &vm->va);
>> >               if (before->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
>> &&
>> > -                 !before->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>> >       } else {
>> >               kfree(before);
>> >       }
>> >
>> >       /* Insert partial mapping after the range */
>> >       if (!list_empty(&after->list)) {
>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(after, &vm->va);
>> >               if (after->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv
>> &&
>> > -                 !after->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>> >       } else {
>> >               kfree(after);
>> >       }
>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev,
>> struct amdgpu_vm *vm, int32_t xcp
>> >       INIT_LIST_HEAD(&vm->evicted);
>> >       INIT_LIST_HEAD(&vm->relocated);
>> >       INIT_LIST_HEAD(&vm->moved);
>> > +     INIT_LIST_HEAD(&vm->dirty);
>> >       INIT_LIST_HEAD(&vm->idle);
>> >       INIT_LIST_HEAD(&vm->invalidated);
>> >       spin_lock_init(&vm->status_lock);
>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >   {
>> >       struct amdgpu_bo_va *bo_va, *tmp;
>> >       u64 total_idle = 0;
>> > +     u64 total_dirty = 0;
>> >       u64 total_relocated = 0;
>> >       u64 total_moved = 0;
>> >       u64 total_invalidated = 0;
>> >       u64 total_done = 0;
>> >       unsigned int total_idle_objs = 0;
>> > +     unsigned int total_dirty_objs = 0;
>> >       unsigned int total_relocated_objs = 0;
>> >       unsigned int total_moved_objs = 0;
>> >       unsigned int total_invalidated_objs = 0;
>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >       total_idle_objs = id;
>> >       id = 0;
>> >
>> > +     seq_puts(m, "\tDirty BOs:\n");
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>> > +             if (!bo_va->base.bo)
>> > +                     continue;
>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo,
>> m);
>> > +     }
>> > +     total_dirty_objs = id;
>> > +     id = 0;
>> > +
>> >       seq_puts(m, "\tRelocated BOs:\n");
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>> base.vm_status) {
>> >               if (!bo_va->base.bo)
>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm
>> *vm, struct seq_file *m)
>> >
>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n",
>> total_idle,
>> >                  total_idle_objs);
>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n",
>> total_dirty,
>> > +                total_dirty_objs);
>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n",
>> total_relocated,
>> >                  total_relocated_objs);
>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n",
>> total_moved,
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > index d9ab97eabda9..f91d4fcf80b8 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>> >       /* per VM BOs moved, but not yet updated in the PT */
>> >       struct list_head        moved;
>> >
>> > +     /* normal and per VM BOs that are not moved, but have new PT
>> entries */
>> > +     struct list_head        dirty;
>> > +
>> >       /* All BOs of this VM not currently in the state machine */
>> >       struct list_head        idle;
>> >
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 22508 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

> On Oct 31, 2023, at 23:17, Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> wrote:
> 
> 
> 
> On Tue, Oct 31, 2023 at 3:08 PM Christian König <christian.koenig@amd.com> wrote:
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>> 
>> 
>> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com> wrote:
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
>> > BO specified in the GEM ioctl; however, when a binding is split, the
>> > adjacent bindings also need to be updated. Such updates currently ends up
>> > getting deferred until next submission which causes stalls.
>> 
>> Yeah, that is a necessity. The hardware simply doesn't support what you 
>> try to do here in all cases.
>> 
>> What can the hardware not do here? Is this just needing to wait for TLB flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older than Polaris) you can't invalidate the TLB while it is in use.
> 
> For Polaris and older it just means that you don't have a guarantee that the shader can't access the memory any more. So delaying the free operation helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to invalidate the TLB while it is in use you can potentially triggering memory accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new VMID for each submission instead of invalidating the old one.
> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as well), but this is something you can really only do on some hw generations after validating that it works.
> 
> I think as long as we make sure all significant work gets done asynchronously, doing the TLB flushing on the next submit (/submissions, one per queue?) is fine for our purposes.

For a bit more of context, the performance / frame timing in Forza with just patch 5 wasn’t quite right. As Bas said, ideally we want to perform all the PT updates right away, and only defer the TLB flush.

For now the state machine part of this patch doesn’t seem to be going in the right direction so I’ll consider dropping this change.

Tatsuyuki.

> 
> (As an aside after thinking some more I *think* we also need some work to make these maps/unmaps (VALID->PRT and PRT->VALID) atomic, as I think it is valid Vulkan to make these race. As such I'm speculating we'd need a bit more reworking there too, not just a late free of the lower level pagetables)
> 
> - Bas 
> 
> Regards,
> Christian. 
> 
>>  
>> 
>> So this approach won't work in general.
>> 
>> Regards,
>> Christian.
>> 
>> >
>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>> > BOs, containing all BOs that have pending updates in `invalids`.
>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
>> > in the dirty state.
>> >
>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> > ---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>> >
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > index a1b15d0d6c48..01d3a97248b0 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>> >    * vital here, so they are not reported back to userspace.
>> >    */
>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> > -                                 struct amdgpu_vm *vm,
>> > -                                 struct amdgpu_bo_va *bo_va,
>> > -                                 uint32_t operation)
>> > +                                 struct amdgpu_vm *vm)
>> >   {
>> > +     struct amdgpu_bo_va *bo_va;
>> >       int r;
>> >   
>> >       if (!amdgpu_vm_ready(vm))
>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> >       if (r)
>> >               goto error;
>> >   
>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>> > +     spin_lock(&vm->status_lock);
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>> >               if (r)
>> >                       goto error;
>> > +             spin_lock(&vm->status_lock);
>> >       }
>> > +     spin_unlock(&vm->status_lock);
>> >   
>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>> >   
>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>> >               break;
>> >       }
>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>> > -                                     args->operation);
>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>> >   
>> >   error:
>> >       drm_exec_fini(&exec);
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > index dd6f72e2a1d6..01d31891cd05 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>> >       spin_unlock(&vm_bo->vm->status_lock);
>> >   }
>> >   
>> > +/**
>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>> > + *
>> > + * @vm_bo: vm_bo which is dirty
>> > + *
>> > + * State for normal and per VM BOs that are not moved, but have new entries in
>> > + * bo_va->invalids.
>> > + */
>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>> > +{
>> > +     spin_lock(&vm_bo->vm->status_lock);
>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>> > +     spin_unlock(&vm_bo->vm->status_lock);
>> > +}
>> > +
>> >   /**
>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>> >    *
>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >   
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>> > +
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >   
>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>> >                       dma_resv_unlock(resv);
>> >               spin_lock(&vm->status_lock);
>> >       }
>> > +
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>> > +             if (r)
>> > +                     return r;
>> > +             spin_lock(&vm->status_lock);
>> > +     }
>> >       spin_unlock(&vm->status_lock);
>> >   
>> >       return 0;
>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>> >                                   struct amdgpu_bo_va_mapping *mapping)
>> >   {
>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>> > -     struct amdgpu_bo *bo = bo_va->base.bo;
>> >   
>> >       mapping->bo_va = bo_va;
>> >       list_add(&mapping->list, &bo_va->invalids);
>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>> > +     if (!bo_va->base.moved)
>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>> >   
>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>> >               amdgpu_vm_prt_get(adev);
>> >   
>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -         !bo_va->base.moved) {
>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>> > -     }
>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>> >   }
>> >   
>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >                       before->flags = tmp->flags;
>> >                       before->bo_va = tmp->bo_va;
>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >   
>> >               /* Remember mapping split at the end */
>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >                       after->flags = tmp->flags;
>> >                       after->bo_va = tmp->bo_va;
>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >   
>> >               list_del(&tmp->list);
>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >   
>> >       /* Insert partial mapping before the range */
>> >       if (!list_empty(&before->list)) {
>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(before, &vm->va);
>> >               if (before->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -                 !before->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>> >       } else {
>> >               kfree(before);
>> >       }
>> >   
>> >       /* Insert partial mapping after the range */
>> >       if (!list_empty(&after->list)) {
>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(after, &vm->va);
>> >               if (after->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -                 !after->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>> >       } else {
>> >               kfree(after);
>> >       }
>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>> >       INIT_LIST_HEAD(&vm->evicted);
>> >       INIT_LIST_HEAD(&vm->relocated);
>> >       INIT_LIST_HEAD(&vm->moved);
>> > +     INIT_LIST_HEAD(&vm->dirty);
>> >       INIT_LIST_HEAD(&vm->idle);
>> >       INIT_LIST_HEAD(&vm->invalidated);
>> >       spin_lock_init(&vm->status_lock);
>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >   {
>> >       struct amdgpu_bo_va *bo_va, *tmp;
>> >       u64 total_idle = 0;
>> > +     u64 total_dirty = 0;
>> >       u64 total_relocated = 0;
>> >       u64 total_moved = 0;
>> >       u64 total_invalidated = 0;
>> >       u64 total_done = 0;
>> >       unsigned int total_idle_objs = 0;
>> > +     unsigned int total_dirty_objs = 0;
>> >       unsigned int total_relocated_objs = 0;
>> >       unsigned int total_moved_objs = 0;
>> >       unsigned int total_invalidated_objs = 0;
>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >       total_idle_objs = id;
>> >       id = 0;
>> >   
>> > +     seq_puts(m, "\tDirty BOs:\n");
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>> > +             if (!bo_va->base.bo)
>> > +                     continue;
>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>> > +     }
>> > +     total_dirty_objs = id;
>> > +     id = 0;
>> > +
>> >       seq_puts(m, "\tRelocated BOs:\n");
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>> >               if (!bo_va->base.bo)
>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >   
>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>> >                  total_idle_objs);
>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
>> > +                total_dirty_objs);
>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>> >                  total_relocated_objs);
>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > index d9ab97eabda9..f91d4fcf80b8 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>> >       /* per VM BOs moved, but not yet updated in the PT */
>> >       struct list_head        moved;
>> >   
>> > +     /* normal and per VM BOs that are not moved, but have new PT entries */
>> > +     struct list_head        dirty;
>> > +
>> >       /* All BOs of this VM not currently in the state machine */
>> >       struct list_head        idle;
>> >   

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

> On Oct 31, 2023, at 23:17, Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl> wrote:
> 
> 
> 
> On Tue, Oct 31, 2023 at 3:08 PM Christian König <christian.koenig@amd.com> wrote:
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>> 
>> 
>> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com> wrote:
>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
>> > BO specified in the GEM ioctl; however, when a binding is split, the
>> > adjacent bindings also need to be updated. Such updates currently ends up
>> > getting deferred until next submission which causes stalls.
>> 
>> Yeah, that is a necessity. The hardware simply doesn't support what you 
>> try to do here in all cases.
>> 
>> What can the hardware not do here? Is this just needing to wait for TLB flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older than Polaris) you can't invalidate the TLB while it is in use.
> 
> For Polaris and older it just means that you don't have a guarantee that the shader can't access the memory any more. So delaying the free operation helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to invalidate the TLB while it is in use you can potentially triggering memory accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new VMID for each submission instead of invalidating the old one.
> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as well), but this is something you can really only do on some hw generations after validating that it works.
> 
> I think as long as we make sure all significant work gets done asynchronously, doing the TLB flushing on the next submit (/submissions, one per queue?) is fine for our purposes.

For a bit more of context, the performance / frame timing in Forza with just patch 5 wasn’t quite right. As Bas said, ideally we want to perform all the PT updates right away, and only defer the TLB flush.

For now the state machine part of this patch doesn’t seem to be going in the right direction so I’ll consider dropping this change.

Tatsuyuki.

> 
> (As an aside after thinking some more I *think* we also need some work to make these maps/unmaps (VALID->PRT and PRT->VALID) atomic, as I think it is valid Vulkan to make these race. As such I'm speculating we'd need a bit more reworking there too, not just a late free of the lower level pagetables)
> 
> - Bas 
> 
> Regards,
> Christian. 
> 
>>  
>> 
>> So this approach won't work in general.
>> 
>> Regards,
>> Christian.
>> 
>> >
>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>> > BOs, containing all BOs that have pending updates in `invalids`.
>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
>> > in the dirty state.
>> >
>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> > ---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>> >
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > index a1b15d0d6c48..01d3a97248b0 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>> >    * vital here, so they are not reported back to userspace.
>> >    */
>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> > -                                 struct amdgpu_vm *vm,
>> > -                                 struct amdgpu_bo_va *bo_va,
>> > -                                 uint32_t operation)
>> > +                                 struct amdgpu_vm *vm)
>> >   {
>> > +     struct amdgpu_bo_va *bo_va;
>> >       int r;
>> >   
>> >       if (!amdgpu_vm_ready(vm))
>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>> >       if (r)
>> >               goto error;
>> >   
>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>> > +     spin_lock(&vm->status_lock);
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>> >               if (r)
>> >                       goto error;
>> > +             spin_lock(&vm->status_lock);
>> >       }
>> > +     spin_unlock(&vm->status_lock);
>> >   
>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>> >   
>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>> >               break;
>> >       }
>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>> > -                                     args->operation);
>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>> >   
>> >   error:
>> >       drm_exec_fini(&exec);
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > index dd6f72e2a1d6..01d31891cd05 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>> >       spin_unlock(&vm_bo->vm->status_lock);
>> >   }
>> >   
>> > +/**
>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>> > + *
>> > + * @vm_bo: vm_bo which is dirty
>> > + *
>> > + * State for normal and per VM BOs that are not moved, but have new entries in
>> > + * bo_va->invalids.
>> > + */
>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>> > +{
>> > +     spin_lock(&vm_bo->vm->status_lock);
>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>> > +     spin_unlock(&vm_bo->vm->status_lock);
>> > +}
>> > +
>> >   /**
>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>> >    *
>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >   
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>> > +
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>> >   
>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>> >                       dma_resv_unlock(resv);
>> >               spin_lock(&vm->status_lock);
>> >       }
>> > +
>> > +     while (!list_empty(&vm->dirty)) {
>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>> > +                                      base.vm_status);
>> > +             spin_unlock(&vm->status_lock);
>> > +
>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>> > +             if (r)
>> > +                     return r;
>> > +             spin_lock(&vm->status_lock);
>> > +     }
>> >       spin_unlock(&vm->status_lock);
>> >   
>> >       return 0;
>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>> >                                   struct amdgpu_bo_va_mapping *mapping)
>> >   {
>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>> > -     struct amdgpu_bo *bo = bo_va->base.bo;
>> >   
>> >       mapping->bo_va = bo_va;
>> >       list_add(&mapping->list, &bo_va->invalids);
>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>> > +     if (!bo_va->base.moved)
>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>> >   
>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>> >               amdgpu_vm_prt_get(adev);
>> >   
>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -         !bo_va->base.moved) {
>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>> > -     }
>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>> >   }
>> >   
>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >                       before->flags = tmp->flags;
>> >                       before->bo_va = tmp->bo_va;
>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >   
>> >               /* Remember mapping split at the end */
>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >                       after->flags = tmp->flags;
>> >                       after->bo_va = tmp->bo_va;
>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>> > +                     if (!tmp->bo_va->base.moved)
>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>> >               }
>> >   
>> >               list_del(&tmp->list);
>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> >   
>> >       /* Insert partial mapping before the range */
>> >       if (!list_empty(&before->list)) {
>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(before, &vm->va);
>> >               if (before->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -                 !before->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>> >       } else {
>> >               kfree(before);
>> >       }
>> >   
>> >       /* Insert partial mapping after the range */
>> >       if (!list_empty(&after->list)) {
>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo;
>> > -
>> >               amdgpu_vm_it_insert(after, &vm->va);
>> >               if (after->flags & AMDGPU_PTE_PRT)
>> >                       amdgpu_vm_prt_get(adev);
>> > -
>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>> > -                 !after->bo_va->base.moved)
>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>> >       } else {
>> >               kfree(after);
>> >       }
>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>> >       INIT_LIST_HEAD(&vm->evicted);
>> >       INIT_LIST_HEAD(&vm->relocated);
>> >       INIT_LIST_HEAD(&vm->moved);
>> > +     INIT_LIST_HEAD(&vm->dirty);
>> >       INIT_LIST_HEAD(&vm->idle);
>> >       INIT_LIST_HEAD(&vm->invalidated);
>> >       spin_lock_init(&vm->status_lock);
>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >   {
>> >       struct amdgpu_bo_va *bo_va, *tmp;
>> >       u64 total_idle = 0;
>> > +     u64 total_dirty = 0;
>> >       u64 total_relocated = 0;
>> >       u64 total_moved = 0;
>> >       u64 total_invalidated = 0;
>> >       u64 total_done = 0;
>> >       unsigned int total_idle_objs = 0;
>> > +     unsigned int total_dirty_objs = 0;
>> >       unsigned int total_relocated_objs = 0;
>> >       unsigned int total_moved_objs = 0;
>> >       unsigned int total_invalidated_objs = 0;
>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >       total_idle_objs = id;
>> >       id = 0;
>> >   
>> > +     seq_puts(m, "\tDirty BOs:\n");
>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>> > +             if (!bo_va->base.bo)
>> > +                     continue;
>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo, m);
>> > +     }
>> > +     total_dirty_objs = id;
>> > +     id = 0;
>> > +
>> >       seq_puts(m, "\tRelocated BOs:\n");
>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>> >               if (!bo_va->base.bo)
>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>> >   
>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>> >                  total_idle_objs);
>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
>> > +                total_dirty_objs);
>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>> >                  total_relocated_objs);
>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > index d9ab97eabda9..f91d4fcf80b8 100644
>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>> >       /* per VM BOs moved, but not yet updated in the PT */
>> >       struct list_head        moved;
>> >   
>> > +     /* normal and per VM BOs that are not moved, but have new PT entries */
>> > +     struct list_head        dirty;
>> > +
>> >       /* All BOs of this VM not currently in the state machine */
>> >       struct list_head        idle;
>> >   

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Lang Yu]

On 10/31/ , Christian König wrote:
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
> > 
> > 
> > On Tue, Oct 31, 2023 at 2:57 PM Christian König
> > <christian.koenig@amd.com> wrote:
> > 
> >     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> >     > The current amdgpu_gem_va_update_vm only tries to perform
> >     updates for the
> >     > BO specified in the GEM ioctl; however, when a binding is split, the
> >     > adjacent bindings also need to be updated. Such updates
> >     currently ends up
> >     > getting deferred until next submission which causes stalls.
> > 
> >     Yeah, that is a necessity. The hardware simply doesn't support
> >     what you
> >     try to do here in all cases.
> > 
> > 
> > What can the hardware not do here? Is this just needing to wait for TLB
> > flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older
> than Polaris) you can't invalidate the TLB while it is in use.

Hi Christian,

non-legacy invalidation can invalidate the TLB while it is in use.
Right? Thanks.

Regards,
Lang

> For Polaris and older it just means that you don't have a guarantee that the
> shader can't access the memory any more. So delaying the free operation
> helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to
> invalidate the TLB while it is in use you can potentially triggering memory
> accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new
> VMID for each submission instead of invalidating the old one.
> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as
> well), but this is something you can really only do on some hw generations
> after validating that it works.
> 
> Regards,
> Christian.
> 
> > 
> > 
> >     So this approach won't work in general.
> > 
> >     Regards,
> >     Christian.
> > 
> >     >
> >     > Introduce a new state "dirty", shared between per-VM BOs and
> >     traditional
> >     > BOs, containing all BOs that have pending updates in `invalids`.
> >     > amdgpu_gem_va_update_vm will now simply flush any pending
> >     updates for BOs
> >     > in the dirty state.
> >     >
> >     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> >     > ---
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
> >     ++++++++++++++++++-------
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
> >     >   3 files changed, 63 insertions(+), 24 deletions(-)
> >     >
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > index a1b15d0d6c48..01d3a97248b0 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
> >     drm_device *dev, void *data,
> >     >    * vital here, so they are not reported back to userspace.
> >     >    */
> >     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> >     > -                                 struct amdgpu_vm *vm,
> >     > -                                 struct amdgpu_bo_va *bo_va,
> >     > -                                 uint32_t operation)
> >     > +                                 struct amdgpu_vm *vm)
> >     >   {
> >     > +     struct amdgpu_bo_va *bo_va;
> >     >       int r;
> >     >
> >     >       if (!amdgpu_vm_ready(vm))
> >     > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
> >     amdgpu_device *adev,
> >     >       if (r)
> >     >               goto error;
> >     >
> >     > -     if (operation == AMDGPU_VA_OP_MAP ||
> >     > -         operation == AMDGPU_VA_OP_REPLACE) {
> >     > +     spin_lock(&vm->status_lock);
> >     > +     while (!list_empty(&vm->dirty)) {
> >     > +             bo_va = list_first_entry(&vm->dirty, struct
> >     amdgpu_bo_va,
> >     > +                                      base.vm_status);
> >     > +             spin_unlock(&vm->status_lock);
> >     > +
> >     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
> >     >               if (r)
> >     >                       goto error;
> >     > +             spin_lock(&vm->status_lock);
> >     >       }
> >     > +     spin_unlock(&vm->status_lock);
> >     >
> >     >       r = amdgpu_vm_update_pdes(adev, vm, false);
> >     >
> >     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
> >     *dev, void *data,
> >     >               break;
> >     >       }
> >     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
> >     !amdgpu_vm_debug)
> >     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> >     > -  args->operation);
> >     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
> >     >
> >     >   error:
> >     >       drm_exec_fini(&exec);
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > index dd6f72e2a1d6..01d31891cd05 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
> >     amdgpu_vm_bo_base *vm_bo, bool evict
> >     >       spin_unlock(&vm_bo->vm->status_lock);
> >     >   }
> >     >
> >     > +/**
> >     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
> >     > + *
> >     > + * @vm_bo: vm_bo which is dirty
> >     > + *
> >     > + * State for normal and per VM BOs that are not moved, but have
> >     new entries in
> >     > + * bo_va->invalids.
> >     > + */
> >     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> >     > +{
> >     > +     spin_lock(&vm_bo->vm->status_lock);
> >     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> >     > +     spin_unlock(&vm_bo->vm->status_lock);
> >     > +}
> >     > +
> >     >   /**
> >     >    * amdgpu_vm_bo_moved - vm_bo is moved
> >     >    *
> >     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
> >     *vm,
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
> >     base.eviction_status)
> >     >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >     >
> >     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
> >     base.vm_status)
> >     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
> >     > +
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> >     base.vm_status)
> >     >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >     >
> >     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
> >     amdgpu_device *adev,
> >     >                       dma_resv_unlock(resv);
> >     >               spin_lock(&vm->status_lock);
> >     >       }
> >     > +
> >     > +     while (!list_empty(&vm->dirty)) {
> >     > +             bo_va = list_first_entry(&vm->dirty, struct
> >     amdgpu_bo_va,
> >     > +                                      base.vm_status);
> >     > +             spin_unlock(&vm->status_lock);
> >     > +
> >     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
> >     > +             if (r)
> >     > +                     return r;
> >     > +             spin_lock(&vm->status_lock);
> >     > +     }
> >     >       spin_unlock(&vm->status_lock);
> >     >
> >     >       return 0;
> >     > @@ -1476,19 +1505,16 @@ static void
> >     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
> >     >                                   struct amdgpu_bo_va_mapping
> >     *mapping)
> >     >   {
> >     >       struct amdgpu_vm *vm = bo_va->base.vm;
> >     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
> >     >
> >     >       mapping->bo_va = bo_va;
> >     >       list_add(&mapping->list, &bo_va->invalids);
> >     >       amdgpu_vm_it_insert(mapping, &vm->va);
> >     > +     if (!bo_va->base.moved)
> >     > +             amdgpu_vm_bo_dirty(&bo_va->base);
> >     >
> >     >       if (mapping->flags & AMDGPU_PTE_PRT)
> >     >               amdgpu_vm_prt_get(adev);
> >     >
> >     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> >     > -         !bo_va->base.moved) {
> >     > -             amdgpu_vm_bo_moved(&bo_va->base);
> >     > -     }
> >     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
> >     >   }
> >     >
> >     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >                       before->flags = tmp->flags;
> >     >                       before->bo_va = tmp->bo_va;
> >     >                       list_add(&before->list,
> >     &tmp->bo_va->invalids);
> >     > +                     if (!tmp->bo_va->base.moved)
> >     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >     >               }
> >     >
> >     >               /* Remember mapping split at the end */
> >     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >                       after->flags = tmp->flags;
> >     >                       after->bo_va = tmp->bo_va;
> >     >                       list_add(&after->list, &tmp->bo_va->invalids);
> >     > +                     if (!tmp->bo_va->base.moved)
> >     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >     >               }
> >     >
> >     >               list_del(&tmp->list);
> >     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >
> >     >       /* Insert partial mapping before the range */
> >     >       if (!list_empty(&before->list)) {
> >     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
> >     <http://base.bo>;
> >     > -
> >     >               amdgpu_vm_it_insert(before, &vm->va);
> >     >               if (before->flags & AMDGPU_PTE_PRT)
> >     >                       amdgpu_vm_prt_get(adev);
> >     > -
> >     > -             if (bo && bo->tbo.base.resv ==
> >     vm->root.bo->tbo.base.resv &&
> >     > -                 !before->bo_va->base.moved)
> >     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
> >     >       } else {
> >     >               kfree(before);
> >     >       }
> >     >
> >     >       /* Insert partial mapping after the range */
> >     >       if (!list_empty(&after->list)) {
> >     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
> >     <http://base.bo>;
> >     > -
> >     >               amdgpu_vm_it_insert(after, &vm->va);
> >     >               if (after->flags & AMDGPU_PTE_PRT)
> >     >                       amdgpu_vm_prt_get(adev);
> >     > -
> >     > -             if (bo && bo->tbo.base.resv ==
> >     vm->root.bo->tbo.base.resv &&
> >     > -                 !after->bo_va->base.moved)
> >     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
> >     >       } else {
> >     >               kfree(after);
> >     >       }
> >     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
> >     *adev, struct amdgpu_vm *vm, int32_t xcp
> >     >       INIT_LIST_HEAD(&vm->evicted);
> >     >       INIT_LIST_HEAD(&vm->relocated);
> >     >       INIT_LIST_HEAD(&vm->moved);
> >     > +     INIT_LIST_HEAD(&vm->dirty);
> >     >       INIT_LIST_HEAD(&vm->idle);
> >     >       INIT_LIST_HEAD(&vm->invalidated);
> >     >       spin_lock_init(&vm->status_lock);
> >     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >   {
> >     >       struct amdgpu_bo_va *bo_va, *tmp;
> >     >       u64 total_idle = 0;
> >     > +     u64 total_dirty = 0;
> >     >       u64 total_relocated = 0;
> >     >       u64 total_moved = 0;
> >     >       u64 total_invalidated = 0;
> >     >       u64 total_done = 0;
> >     >       unsigned int total_idle_objs = 0;
> >     > +     unsigned int total_dirty_objs = 0;
> >     >       unsigned int total_relocated_objs = 0;
> >     >       unsigned int total_moved_objs = 0;
> >     >       unsigned int total_invalidated_objs = 0;
> >     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >       total_idle_objs = id;
> >     >       id = 0;
> >     >
> >     > +     seq_puts(m, "\tDirty BOs:\n");
> >     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
> >     base.vm_status) {
> >     > +             if (!bo_va->base.bo <http://base.bo>)
> >     > +                     continue;
> >     > +             total_dirty += amdgpu_bo_print_info(id++,
> >     bo_va->base.bo <http://base.bo>, m);
> >     > +     }
> >     > +     total_dirty_objs = id;
> >     > +     id = 0;
> >     > +
> >     >       seq_puts(m, "\tRelocated BOs:\n");
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> >     base.vm_status) {
> >     >               if (!bo_va->base.bo <http://base.bo>)
> >     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >
> >     >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
> >     total_idle,
> >     >                  total_idle_objs);
> >     > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
> >     total_dirty,
> >     > +                total_dirty_objs);
> >     >       seq_printf(m, "\tTotal relocated size:
> >      %12lld\tobjs:\t%d\n", total_relocated,
> >     >                  total_relocated_objs);
> >     >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
> >     total_moved,
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > index d9ab97eabda9..f91d4fcf80b8 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
> >     >       /* per VM BOs moved, but not yet updated in the PT */
> >     >       struct list_head        moved;
> >     >
> >     > +     /* normal and per VM BOs that are not moved, but have new
> >     PT entries */
> >     > +     struct list_head        dirty;
> >     > +
> >     >       /* All BOs of this VM not currently in the state machine */
> >     >       struct list_head        idle;
> >     >
> > 

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Lang Yu]

On 10/31/ , Christian König wrote:
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
> > 
> > 
> > On Tue, Oct 31, 2023 at 2:57 PM Christian König
> > <christian.koenig@amd.com> wrote:
> > 
> >     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> >     > The current amdgpu_gem_va_update_vm only tries to perform
> >     updates for the
> >     > BO specified in the GEM ioctl; however, when a binding is split, the
> >     > adjacent bindings also need to be updated. Such updates
> >     currently ends up
> >     > getting deferred until next submission which causes stalls.
> > 
> >     Yeah, that is a necessity. The hardware simply doesn't support
> >     what you
> >     try to do here in all cases.
> > 
> > 
> > What can the hardware not do here? Is this just needing to wait for TLB
> > flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older
> than Polaris) you can't invalidate the TLB while it is in use.

Hi Christian,

non-legacy invalidation can invalidate the TLB while it is in use.
Right? Thanks.

Regards,
Lang

> For Polaris and older it just means that you don't have a guarantee that the
> shader can't access the memory any more. So delaying the free operation
> helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to
> invalidate the TLB while it is in use you can potentially triggering memory
> accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new
> VMID for each submission instead of invalidating the old one.
> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as
> well), but this is something you can really only do on some hw generations
> after validating that it works.
> 
> Regards,
> Christian.
> 
> > 
> > 
> >     So this approach won't work in general.
> > 
> >     Regards,
> >     Christian.
> > 
> >     >
> >     > Introduce a new state "dirty", shared between per-VM BOs and
> >     traditional
> >     > BOs, containing all BOs that have pending updates in `invalids`.
> >     > amdgpu_gem_va_update_vm will now simply flush any pending
> >     updates for BOs
> >     > in the dirty state.
> >     >
> >     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> >     > ---
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
> >     ++++++++++++++++++-------
> >     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
> >     >   3 files changed, 63 insertions(+), 24 deletions(-)
> >     >
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > index a1b15d0d6c48..01d3a97248b0 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> >     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
> >     drm_device *dev, void *data,
> >     >    * vital here, so they are not reported back to userspace.
> >     >    */
> >     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
> >     > -                                 struct amdgpu_vm *vm,
> >     > -                                 struct amdgpu_bo_va *bo_va,
> >     > -                                 uint32_t operation)
> >     > +                                 struct amdgpu_vm *vm)
> >     >   {
> >     > +     struct amdgpu_bo_va *bo_va;
> >     >       int r;
> >     >
> >     >       if (!amdgpu_vm_ready(vm))
> >     > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
> >     amdgpu_device *adev,
> >     >       if (r)
> >     >               goto error;
> >     >
> >     > -     if (operation == AMDGPU_VA_OP_MAP ||
> >     > -         operation == AMDGPU_VA_OP_REPLACE) {
> >     > +     spin_lock(&vm->status_lock);
> >     > +     while (!list_empty(&vm->dirty)) {
> >     > +             bo_va = list_first_entry(&vm->dirty, struct
> >     amdgpu_bo_va,
> >     > +                                      base.vm_status);
> >     > +             spin_unlock(&vm->status_lock);
> >     > +
> >     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
> >     >               if (r)
> >     >                       goto error;
> >     > +             spin_lock(&vm->status_lock);
> >     >       }
> >     > +     spin_unlock(&vm->status_lock);
> >     >
> >     >       r = amdgpu_vm_update_pdes(adev, vm, false);
> >     >
> >     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
> >     *dev, void *data,
> >     >               break;
> >     >       }
> >     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
> >     !amdgpu_vm_debug)
> >     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
> >     > -  args->operation);
> >     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
> >     >
> >     >   error:
> >     >       drm_exec_fini(&exec);
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > index dd6f72e2a1d6..01d31891cd05 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> >     > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
> >     amdgpu_vm_bo_base *vm_bo, bool evict
> >     >       spin_unlock(&vm_bo->vm->status_lock);
> >     >   }
> >     >
> >     > +/**
> >     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
> >     > + *
> >     > + * @vm_bo: vm_bo which is dirty
> >     > + *
> >     > + * State for normal and per VM BOs that are not moved, but have
> >     new entries in
> >     > + * bo_va->invalids.
> >     > + */
> >     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
> >     > +{
> >     > +     spin_lock(&vm_bo->vm->status_lock);
> >     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
> >     > +     spin_unlock(&vm_bo->vm->status_lock);
> >     > +}
> >     > +
> >     >   /**
> >     >    * amdgpu_vm_bo_moved - vm_bo is moved
> >     >    *
> >     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
> >     *vm,
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
> >     base.eviction_status)
> >     >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >     >
> >     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
> >     base.vm_status)
> >     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
> >     > +
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> >     base.vm_status)
> >     >               amdgpu_vm_bo_get_memory(bo_va, stats);
> >     >
> >     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
> >     amdgpu_device *adev,
> >     >                       dma_resv_unlock(resv);
> >     >               spin_lock(&vm->status_lock);
> >     >       }
> >     > +
> >     > +     while (!list_empty(&vm->dirty)) {
> >     > +             bo_va = list_first_entry(&vm->dirty, struct
> >     amdgpu_bo_va,
> >     > +                                      base.vm_status);
> >     > +             spin_unlock(&vm->status_lock);
> >     > +
> >     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
> >     > +             if (r)
> >     > +                     return r;
> >     > +             spin_lock(&vm->status_lock);
> >     > +     }
> >     >       spin_unlock(&vm->status_lock);
> >     >
> >     >       return 0;
> >     > @@ -1476,19 +1505,16 @@ static void
> >     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
> >     >                                   struct amdgpu_bo_va_mapping
> >     *mapping)
> >     >   {
> >     >       struct amdgpu_vm *vm = bo_va->base.vm;
> >     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
> >     >
> >     >       mapping->bo_va = bo_va;
> >     >       list_add(&mapping->list, &bo_va->invalids);
> >     >       amdgpu_vm_it_insert(mapping, &vm->va);
> >     > +     if (!bo_va->base.moved)
> >     > +             amdgpu_vm_bo_dirty(&bo_va->base);
> >     >
> >     >       if (mapping->flags & AMDGPU_PTE_PRT)
> >     >               amdgpu_vm_prt_get(adev);
> >     >
> >     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
> >     > -         !bo_va->base.moved) {
> >     > -             amdgpu_vm_bo_moved(&bo_va->base);
> >     > -     }
> >     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
> >     >   }
> >     >
> >     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >                       before->flags = tmp->flags;
> >     >                       before->bo_va = tmp->bo_va;
> >     >                       list_add(&before->list,
> >     &tmp->bo_va->invalids);
> >     > +                     if (!tmp->bo_va->base.moved)
> >     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >     >               }
> >     >
> >     >               /* Remember mapping split at the end */
> >     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >                       after->flags = tmp->flags;
> >     >                       after->bo_va = tmp->bo_va;
> >     >                       list_add(&after->list, &tmp->bo_va->invalids);
> >     > +                     if (!tmp->bo_va->base.moved)
> >     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
> >     >               }
> >     >
> >     >               list_del(&tmp->list);
> >     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
> >     amdgpu_device *adev,
> >     >
> >     >       /* Insert partial mapping before the range */
> >     >       if (!list_empty(&before->list)) {
> >     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
> >     <http://base.bo>;
> >     > -
> >     >               amdgpu_vm_it_insert(before, &vm->va);
> >     >               if (before->flags & AMDGPU_PTE_PRT)
> >     >                       amdgpu_vm_prt_get(adev);
> >     > -
> >     > -             if (bo && bo->tbo.base.resv ==
> >     vm->root.bo->tbo.base.resv &&
> >     > -                 !before->bo_va->base.moved)
> >     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
> >     >       } else {
> >     >               kfree(before);
> >     >       }
> >     >
> >     >       /* Insert partial mapping after the range */
> >     >       if (!list_empty(&after->list)) {
> >     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
> >     <http://base.bo>;
> >     > -
> >     >               amdgpu_vm_it_insert(after, &vm->va);
> >     >               if (after->flags & AMDGPU_PTE_PRT)
> >     >                       amdgpu_vm_prt_get(adev);
> >     > -
> >     > -             if (bo && bo->tbo.base.resv ==
> >     vm->root.bo->tbo.base.resv &&
> >     > -                 !after->bo_va->base.moved)
> >     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
> >     >       } else {
> >     >               kfree(after);
> >     >       }
> >     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
> >     *adev, struct amdgpu_vm *vm, int32_t xcp
> >     >       INIT_LIST_HEAD(&vm->evicted);
> >     >       INIT_LIST_HEAD(&vm->relocated);
> >     >       INIT_LIST_HEAD(&vm->moved);
> >     > +     INIT_LIST_HEAD(&vm->dirty);
> >     >       INIT_LIST_HEAD(&vm->idle);
> >     >       INIT_LIST_HEAD(&vm->invalidated);
> >     >       spin_lock_init(&vm->status_lock);
> >     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >   {
> >     >       struct amdgpu_bo_va *bo_va, *tmp;
> >     >       u64 total_idle = 0;
> >     > +     u64 total_dirty = 0;
> >     >       u64 total_relocated = 0;
> >     >       u64 total_moved = 0;
> >     >       u64 total_invalidated = 0;
> >     >       u64 total_done = 0;
> >     >       unsigned int total_idle_objs = 0;
> >     > +     unsigned int total_dirty_objs = 0;
> >     >       unsigned int total_relocated_objs = 0;
> >     >       unsigned int total_moved_objs = 0;
> >     >       unsigned int total_invalidated_objs = 0;
> >     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >       total_idle_objs = id;
> >     >       id = 0;
> >     >
> >     > +     seq_puts(m, "\tDirty BOs:\n");
> >     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
> >     base.vm_status) {
> >     > +             if (!bo_va->base.bo <http://base.bo>)
> >     > +                     continue;
> >     > +             total_dirty += amdgpu_bo_print_info(id++,
> >     bo_va->base.bo <http://base.bo>, m);
> >     > +     }
> >     > +     total_dirty_objs = id;
> >     > +     id = 0;
> >     > +
> >     >       seq_puts(m, "\tRelocated BOs:\n");
> >     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
> >     base.vm_status) {
> >     >               if (!bo_va->base.bo <http://base.bo>)
> >     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
> >     amdgpu_vm *vm, struct seq_file *m)
> >     >
> >     >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
> >     total_idle,
> >     >                  total_idle_objs);
> >     > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
> >     total_dirty,
> >     > +                total_dirty_objs);
> >     >       seq_printf(m, "\tTotal relocated size:
> >      %12lld\tobjs:\t%d\n", total_relocated,
> >     >                  total_relocated_objs);
> >     >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
> >     total_moved,
> >     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > index d9ab97eabda9..f91d4fcf80b8 100644
> >     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
> >     >       /* per VM BOs moved, but not yet updated in the PT */
> >     >       struct list_head        moved;
> >     >
> >     > +     /* normal and per VM BOs that are not moved, but have new
> >     PT entries */
> >     > +     struct list_head        dirty;
> >     > +
> >     >       /* All BOs of this VM not currently in the state machine */
> >     >       struct list_head        idle;
> >     >
> > 

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

Am 02.11.23 um 03:36 schrieb Lang Yu:
> On 10/31/ , Christian König wrote:
>> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>>>
>>> On Tue, Oct 31, 2023 at 2:57 PM Christian König
>>> <christian.koenig@amd.com> wrote:
>>>
>>>      Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>>      > The current amdgpu_gem_va_update_vm only tries to perform
>>>      updates for the
>>>      > BO specified in the GEM ioctl; however, when a binding is split, the
>>>      > adjacent bindings also need to be updated. Such updates
>>>      currently ends up
>>>      > getting deferred until next submission which causes stalls.
>>>
>>>      Yeah, that is a necessity. The hardware simply doesn't support
>>>      what you
>>>      try to do here in all cases.
>>>
>>>
>>> What can the hardware not do here? Is this just needing to wait for TLB
>>> flushes before we can free pagetables, can we just delay that?
>> On some hardware generations (especially Navi1x, but also everything older
>> than Polaris) you can't invalidate the TLB while it is in use.
> Hi Christian,
>
> non-legacy invalidation can invalidate the TLB while it is in use.
> Right? Thanks.

Right, the problem is that they are only available starting with Vega 
(for GFX8 they only work for the APUs IIRC).

Regards,
Christian.

>
> Regards,
> Lang
>
>> For Polaris and older it just means that you don't have a guarantee that the
>> shader can't access the memory any more. So delaying the free operation
>> helps here.
>>
>> But for Navi1x it's a workaround for a hardware bug. If you try to
>> invalidate the TLB while it is in use you can potentially triggering memory
>> accesses to random addresses.
>>
>> That's why we still delay TLB invalidation's to the next CS and use a new
>> VMID for each submission instead of invalidating the old one.
>>
>> I'm currently working on changing that for Navi2x and newer (maybe Vega as
>> well), but this is something you can really only do on some hw generations
>> after validating that it works.
>>
>> Regards,
>> Christian.
>>
>>>
>>>      So this approach won't work in general.
>>>
>>>      Regards,
>>>      Christian.
>>>
>>>      >
>>>      > Introduce a new state "dirty", shared between per-VM BOs and
>>>      traditional
>>>      > BOs, containing all BOs that have pending updates in `invalids`.
>>>      > amdgpu_gem_va_update_vm will now simply flush any pending
>>>      updates for BOs
>>>      > in the dirty state.
>>>      >
>>>      > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>>      > ---
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>>>      ++++++++++++++++++-------
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>>>      >   3 files changed, 63 insertions(+), 24 deletions(-)
>>>      >
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > index a1b15d0d6c48..01d3a97248b0 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>>>      drm_device *dev, void *data,
>>>      >    * vital here, so they are not reported back to userspace.
>>>      >    */
>>>      >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>      > -                                 struct amdgpu_vm *vm,
>>>      > -                                 struct amdgpu_bo_va *bo_va,
>>>      > -                                 uint32_t operation)
>>>      > +                                 struct amdgpu_vm *vm)
>>>      >   {
>>>      > +     struct amdgpu_bo_va *bo_va;
>>>      >       int r;
>>>      >
>>>      >       if (!amdgpu_vm_ready(vm))
>>>      > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>>>      amdgpu_device *adev,
>>>      >       if (r)
>>>      >               goto error;
>>>      >
>>>      > -     if (operation == AMDGPU_VA_OP_MAP ||
>>>      > -         operation == AMDGPU_VA_OP_REPLACE) {
>>>      > +     spin_lock(&vm->status_lock);
>>>      > +     while (!list_empty(&vm->dirty)) {
>>>      > +             bo_va = list_first_entry(&vm->dirty, struct
>>>      amdgpu_bo_va,
>>>      > +                                      base.vm_status);
>>>      > +             spin_unlock(&vm->status_lock);
>>>      > +
>>>      >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>      >               if (r)
>>>      >                       goto error;
>>>      > +             spin_lock(&vm->status_lock);
>>>      >       }
>>>      > +     spin_unlock(&vm->status_lock);
>>>      >
>>>      >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>>      >
>>>      > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>>>      *dev, void *data,
>>>      >               break;
>>>      >       }
>>>      >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>>>      !amdgpu_vm_debug)
>>>      > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>>      > -  args->operation);
>>>      > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>>      >
>>>      >   error:
>>>      >       drm_exec_fini(&exec);
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > index dd6f72e2a1d6..01d31891cd05 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>>>      amdgpu_vm_bo_base *vm_bo, bool evict
>>>      >       spin_unlock(&vm_bo->vm->status_lock);
>>>      >   }
>>>      >
>>>      > +/**
>>>      > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>>      > + *
>>>      > + * @vm_bo: vm_bo which is dirty
>>>      > + *
>>>      > + * State for normal and per VM BOs that are not moved, but have
>>>      new entries in
>>>      > + * bo_va->invalids.
>>>      > + */
>>>      > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>>      > +{
>>>      > +     spin_lock(&vm_bo->vm->status_lock);
>>>      > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>>      > +     spin_unlock(&vm_bo->vm->status_lock);
>>>      > +}
>>>      > +
>>>      >   /**
>>>      >    * amdgpu_vm_bo_moved - vm_bo is moved
>>>      >    *
>>>      > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
>>>      *vm,
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>>>      base.eviction_status)
>>>      >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      >
>>>      > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>      base.vm_status)
>>>      > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      > +
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>      base.vm_status)
>>>      >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      >
>>>      > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>>>      amdgpu_device *adev,
>>>      >                       dma_resv_unlock(resv);
>>>      >               spin_lock(&vm->status_lock);
>>>      >       }
>>>      > +
>>>      > +     while (!list_empty(&vm->dirty)) {
>>>      > +             bo_va = list_first_entry(&vm->dirty, struct
>>>      amdgpu_bo_va,
>>>      > +                                      base.vm_status);
>>>      > +             spin_unlock(&vm->status_lock);
>>>      > +
>>>      > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>      > +             if (r)
>>>      > +                     return r;
>>>      > +             spin_lock(&vm->status_lock);
>>>      > +     }
>>>      >       spin_unlock(&vm->status_lock);
>>>      >
>>>      >       return 0;
>>>      > @@ -1476,19 +1505,16 @@ static void
>>>      amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>>      >                                   struct amdgpu_bo_va_mapping
>>>      *mapping)
>>>      >   {
>>>      >       struct amdgpu_vm *vm = bo_va->base.vm;
>>>      > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
>>>      >
>>>      >       mapping->bo_va = bo_va;
>>>      >       list_add(&mapping->list, &bo_va->invalids);
>>>      >       amdgpu_vm_it_insert(mapping, &vm->va);
>>>      > +     if (!bo_va->base.moved)
>>>      > +             amdgpu_vm_bo_dirty(&bo_va->base);
>>>      >
>>>      >       if (mapping->flags & AMDGPU_PTE_PRT)
>>>      >               amdgpu_vm_prt_get(adev);
>>>      >
>>>      > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>>      > -         !bo_va->base.moved) {
>>>      > -             amdgpu_vm_bo_moved(&bo_va->base);
>>>      > -     }
>>>      >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>>      >   }
>>>      >
>>>      > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >                       before->flags = tmp->flags;
>>>      >                       before->bo_va = tmp->bo_va;
>>>      >                       list_add(&before->list,
>>>      &tmp->bo_va->invalids);
>>>      > +                     if (!tmp->bo_va->base.moved)
>>>      > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>      >               }
>>>      >
>>>      >               /* Remember mapping split at the end */
>>>      > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >                       after->flags = tmp->flags;
>>>      >                       after->bo_va = tmp->bo_va;
>>>      >                       list_add(&after->list, &tmp->bo_va->invalids);
>>>      > +                     if (!tmp->bo_va->base.moved)
>>>      > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>      >               }
>>>      >
>>>      >               list_del(&tmp->list);
>>>      > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >
>>>      >       /* Insert partial mapping before the range */
>>>      >       if (!list_empty(&before->list)) {
>>>      > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>>>      <http://base.bo>;
>>>      > -
>>>      >               amdgpu_vm_it_insert(before, &vm->va);
>>>      >               if (before->flags & AMDGPU_PTE_PRT)
>>>      >                       amdgpu_vm_prt_get(adev);
>>>      > -
>>>      > -             if (bo && bo->tbo.base.resv ==
>>>      vm->root.bo->tbo.base.resv &&
>>>      > -                 !before->bo_va->base.moved)
>>>      > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>>>      >       } else {
>>>      >               kfree(before);
>>>      >       }
>>>      >
>>>      >       /* Insert partial mapping after the range */
>>>      >       if (!list_empty(&after->list)) {
>>>      > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>>>      <http://base.bo>;
>>>      > -
>>>      >               amdgpu_vm_it_insert(after, &vm->va);
>>>      >               if (after->flags & AMDGPU_PTE_PRT)
>>>      >                       amdgpu_vm_prt_get(adev);
>>>      > -
>>>      > -             if (bo && bo->tbo.base.resv ==
>>>      vm->root.bo->tbo.base.resv &&
>>>      > -                 !after->bo_va->base.moved)
>>>      > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>>>      >       } else {
>>>      >               kfree(after);
>>>      >       }
>>>      > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>>>      *adev, struct amdgpu_vm *vm, int32_t xcp
>>>      >       INIT_LIST_HEAD(&vm->evicted);
>>>      >       INIT_LIST_HEAD(&vm->relocated);
>>>      >       INIT_LIST_HEAD(&vm->moved);
>>>      > +     INIT_LIST_HEAD(&vm->dirty);
>>>      >       INIT_LIST_HEAD(&vm->idle);
>>>      >       INIT_LIST_HEAD(&vm->invalidated);
>>>      >       spin_lock_init(&vm->status_lock);
>>>      > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >   {
>>>      >       struct amdgpu_bo_va *bo_va, *tmp;
>>>      >       u64 total_idle = 0;
>>>      > +     u64 total_dirty = 0;
>>>      >       u64 total_relocated = 0;
>>>      >       u64 total_moved = 0;
>>>      >       u64 total_invalidated = 0;
>>>      >       u64 total_done = 0;
>>>      >       unsigned int total_idle_objs = 0;
>>>      > +     unsigned int total_dirty_objs = 0;
>>>      >       unsigned int total_relocated_objs = 0;
>>>      >       unsigned int total_moved_objs = 0;
>>>      >       unsigned int total_invalidated_objs = 0;
>>>      > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >       total_idle_objs = id;
>>>      >       id = 0;
>>>      >
>>>      > +     seq_puts(m, "\tDirty BOs:\n");
>>>      > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>      base.vm_status) {
>>>      > +             if (!bo_va->base.bo <http://base.bo>)
>>>      > +                     continue;
>>>      > +             total_dirty += amdgpu_bo_print_info(id++,
>>>      bo_va->base.bo <http://base.bo>, m);
>>>      > +     }
>>>      > +     total_dirty_objs = id;
>>>      > +     id = 0;
>>>      > +
>>>      >       seq_puts(m, "\tRelocated BOs:\n");
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>      base.vm_status) {
>>>      >               if (!bo_va->base.bo <http://base.bo>)
>>>      > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >
>>>      >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
>>>      total_idle,
>>>      >                  total_idle_objs);
>>>      > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
>>>      total_dirty,
>>>      > +                total_dirty_objs);
>>>      >       seq_printf(m, "\tTotal relocated size:
>>>       %12lld\tobjs:\t%d\n", total_relocated,
>>>      >                  total_relocated_objs);
>>>      >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
>>>      total_moved,
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > index d9ab97eabda9..f91d4fcf80b8 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>>      >       /* per VM BOs moved, but not yet updated in the PT */
>>>      >       struct list_head        moved;
>>>      >
>>>      > +     /* normal and per VM BOs that are not moved, but have new
>>>      PT entries */
>>>      > +     struct list_head        dirty;
>>>      > +
>>>      >       /* All BOs of this VM not currently in the state machine */
>>>      >       struct list_head        idle;
>>>      >
>>>

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

Am 02.11.23 um 03:36 schrieb Lang Yu:
> On 10/31/ , Christian König wrote:
>> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>>>
>>> On Tue, Oct 31, 2023 at 2:57 PM Christian König
>>> <christian.koenig@amd.com> wrote:
>>>
>>>      Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>>      > The current amdgpu_gem_va_update_vm only tries to perform
>>>      updates for the
>>>      > BO specified in the GEM ioctl; however, when a binding is split, the
>>>      > adjacent bindings also need to be updated. Such updates
>>>      currently ends up
>>>      > getting deferred until next submission which causes stalls.
>>>
>>>      Yeah, that is a necessity. The hardware simply doesn't support
>>>      what you
>>>      try to do here in all cases.
>>>
>>>
>>> What can the hardware not do here? Is this just needing to wait for TLB
>>> flushes before we can free pagetables, can we just delay that?
>> On some hardware generations (especially Navi1x, but also everything older
>> than Polaris) you can't invalidate the TLB while it is in use.
> Hi Christian,
>
> non-legacy invalidation can invalidate the TLB while it is in use.
> Right? Thanks.

Right, the problem is that they are only available starting with Vega 
(for GFX8 they only work for the APUs IIRC).

Regards,
Christian.

>
> Regards,
> Lang
>
>> For Polaris and older it just means that you don't have a guarantee that the
>> shader can't access the memory any more. So delaying the free operation
>> helps here.
>>
>> But for Navi1x it's a workaround for a hardware bug. If you try to
>> invalidate the TLB while it is in use you can potentially triggering memory
>> accesses to random addresses.
>>
>> That's why we still delay TLB invalidation's to the next CS and use a new
>> VMID for each submission instead of invalidating the old one.
>>
>> I'm currently working on changing that for Navi2x and newer (maybe Vega as
>> well), but this is something you can really only do on some hw generations
>> after validating that it works.
>>
>> Regards,
>> Christian.
>>
>>>
>>>      So this approach won't work in general.
>>>
>>>      Regards,
>>>      Christian.
>>>
>>>      >
>>>      > Introduce a new state "dirty", shared between per-VM BOs and
>>>      traditional
>>>      > BOs, containing all BOs that have pending updates in `invalids`.
>>>      > amdgpu_gem_va_update_vm will now simply flush any pending
>>>      updates for BOs
>>>      > in the dirty state.
>>>      >
>>>      > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>>      > ---
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>>>      ++++++++++++++++++-------
>>>      >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>>>      >   3 files changed, 63 insertions(+), 24 deletions(-)
>>>      >
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > index a1b15d0d6c48..01d3a97248b0 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>      > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>>>      drm_device *dev, void *data,
>>>      >    * vital here, so they are not reported back to userspace.
>>>      >    */
>>>      >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>      > -                                 struct amdgpu_vm *vm,
>>>      > -                                 struct amdgpu_bo_va *bo_va,
>>>      > -                                 uint32_t operation)
>>>      > +                                 struct amdgpu_vm *vm)
>>>      >   {
>>>      > +     struct amdgpu_bo_va *bo_va;
>>>      >       int r;
>>>      >
>>>      >       if (!amdgpu_vm_ready(vm))
>>>      > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct
>>>      amdgpu_device *adev,
>>>      >       if (r)
>>>      >               goto error;
>>>      >
>>>      > -     if (operation == AMDGPU_VA_OP_MAP ||
>>>      > -         operation == AMDGPU_VA_OP_REPLACE) {
>>>      > +     spin_lock(&vm->status_lock);
>>>      > +     while (!list_empty(&vm->dirty)) {
>>>      > +             bo_va = list_first_entry(&vm->dirty, struct
>>>      amdgpu_bo_va,
>>>      > +                                      base.vm_status);
>>>      > +             spin_unlock(&vm->status_lock);
>>>      > +
>>>      >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>      >               if (r)
>>>      >                       goto error;
>>>      > +             spin_lock(&vm->status_lock);
>>>      >       }
>>>      > +     spin_unlock(&vm->status_lock);
>>>      >
>>>      >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>>      >
>>>      > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>>>      *dev, void *data,
>>>      >               break;
>>>      >       }
>>>      >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>>>      !amdgpu_vm_debug)
>>>      > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>>      > -  args->operation);
>>>      > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>>      >
>>>      >   error:
>>>      >       drm_exec_fini(&exec);
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > index dd6f72e2a1d6..01d31891cd05 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>      > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct
>>>      amdgpu_vm_bo_base *vm_bo, bool evict
>>>      >       spin_unlock(&vm_bo->vm->status_lock);
>>>      >   }
>>>      >
>>>      > +/**
>>>      > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>>      > + *
>>>      > + * @vm_bo: vm_bo which is dirty
>>>      > + *
>>>      > + * State for normal and per VM BOs that are not moved, but have
>>>      new entries in
>>>      > + * bo_va->invalids.
>>>      > + */
>>>      > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>>      > +{
>>>      > +     spin_lock(&vm_bo->vm->status_lock);
>>>      > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>>      > +     spin_unlock(&vm_bo->vm->status_lock);
>>>      > +}
>>>      > +
>>>      >   /**
>>>      >    * amdgpu_vm_bo_moved - vm_bo is moved
>>>      >    *
>>>      > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm
>>>      *vm,
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>>>      base.eviction_status)
>>>      >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      >
>>>      > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>      base.vm_status)
>>>      > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      > +
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>      base.vm_status)
>>>      >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>      >
>>>      > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>>>      amdgpu_device *adev,
>>>      >                       dma_resv_unlock(resv);
>>>      >               spin_lock(&vm->status_lock);
>>>      >       }
>>>      > +
>>>      > +     while (!list_empty(&vm->dirty)) {
>>>      > +             bo_va = list_first_entry(&vm->dirty, struct
>>>      amdgpu_bo_va,
>>>      > +                                      base.vm_status);
>>>      > +             spin_unlock(&vm->status_lock);
>>>      > +
>>>      > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>      > +             if (r)
>>>      > +                     return r;
>>>      > +             spin_lock(&vm->status_lock);
>>>      > +     }
>>>      >       spin_unlock(&vm->status_lock);
>>>      >
>>>      >       return 0;
>>>      > @@ -1476,19 +1505,16 @@ static void
>>>      amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>>      >                                   struct amdgpu_bo_va_mapping
>>>      *mapping)
>>>      >   {
>>>      >       struct amdgpu_vm *vm = bo_va->base.vm;
>>>      > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo>;
>>>      >
>>>      >       mapping->bo_va = bo_va;
>>>      >       list_add(&mapping->list, &bo_va->invalids);
>>>      >       amdgpu_vm_it_insert(mapping, &vm->va);
>>>      > +     if (!bo_va->base.moved)
>>>      > +             amdgpu_vm_bo_dirty(&bo_va->base);
>>>      >
>>>      >       if (mapping->flags & AMDGPU_PTE_PRT)
>>>      >               amdgpu_vm_prt_get(adev);
>>>      >
>>>      > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>>      > -         !bo_va->base.moved) {
>>>      > -             amdgpu_vm_bo_moved(&bo_va->base);
>>>      > -     }
>>>      >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>>      >   }
>>>      >
>>>      > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >                       before->flags = tmp->flags;
>>>      >                       before->bo_va = tmp->bo_va;
>>>      >                       list_add(&before->list,
>>>      &tmp->bo_va->invalids);
>>>      > +                     if (!tmp->bo_va->base.moved)
>>>      > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>      >               }
>>>      >
>>>      >               /* Remember mapping split at the end */
>>>      > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >                       after->flags = tmp->flags;
>>>      >                       after->bo_va = tmp->bo_va;
>>>      >                       list_add(&after->list, &tmp->bo_va->invalids);
>>>      > +                     if (!tmp->bo_va->base.moved)
>>>      > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>      >               }
>>>      >
>>>      >               list_del(&tmp->list);
>>>      > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>      amdgpu_device *adev,
>>>      >
>>>      >       /* Insert partial mapping before the range */
>>>      >       if (!list_empty(&before->list)) {
>>>      > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>>>      <http://base.bo>;
>>>      > -
>>>      >               amdgpu_vm_it_insert(before, &vm->va);
>>>      >               if (before->flags & AMDGPU_PTE_PRT)
>>>      >                       amdgpu_vm_prt_get(adev);
>>>      > -
>>>      > -             if (bo && bo->tbo.base.resv ==
>>>      vm->root.bo->tbo.base.resv &&
>>>      > -                 !before->bo_va->base.moved)
>>>      > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>>>      >       } else {
>>>      >               kfree(before);
>>>      >       }
>>>      >
>>>      >       /* Insert partial mapping after the range */
>>>      >       if (!list_empty(&after->list)) {
>>>      > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>>>      <http://base.bo>;
>>>      > -
>>>      >               amdgpu_vm_it_insert(after, &vm->va);
>>>      >               if (after->flags & AMDGPU_PTE_PRT)
>>>      >                       amdgpu_vm_prt_get(adev);
>>>      > -
>>>      > -             if (bo && bo->tbo.base.resv ==
>>>      vm->root.bo->tbo.base.resv &&
>>>      > -                 !after->bo_va->base.moved)
>>>      > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>>>      >       } else {
>>>      >               kfree(after);
>>>      >       }
>>>      > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>>>      *adev, struct amdgpu_vm *vm, int32_t xcp
>>>      >       INIT_LIST_HEAD(&vm->evicted);
>>>      >       INIT_LIST_HEAD(&vm->relocated);
>>>      >       INIT_LIST_HEAD(&vm->moved);
>>>      > +     INIT_LIST_HEAD(&vm->dirty);
>>>      >       INIT_LIST_HEAD(&vm->idle);
>>>      >       INIT_LIST_HEAD(&vm->invalidated);
>>>      >       spin_lock_init(&vm->status_lock);
>>>      > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >   {
>>>      >       struct amdgpu_bo_va *bo_va, *tmp;
>>>      >       u64 total_idle = 0;
>>>      > +     u64 total_dirty = 0;
>>>      >       u64 total_relocated = 0;
>>>      >       u64 total_moved = 0;
>>>      >       u64 total_invalidated = 0;
>>>      >       u64 total_done = 0;
>>>      >       unsigned int total_idle_objs = 0;
>>>      > +     unsigned int total_dirty_objs = 0;
>>>      >       unsigned int total_relocated_objs = 0;
>>>      >       unsigned int total_moved_objs = 0;
>>>      >       unsigned int total_invalidated_objs = 0;
>>>      > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >       total_idle_objs = id;
>>>      >       id = 0;
>>>      >
>>>      > +     seq_puts(m, "\tDirty BOs:\n");
>>>      > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>      base.vm_status) {
>>>      > +             if (!bo_va->base.bo <http://base.bo>)
>>>      > +                     continue;
>>>      > +             total_dirty += amdgpu_bo_print_info(id++,
>>>      bo_va->base.bo <http://base.bo>, m);
>>>      > +     }
>>>      > +     total_dirty_objs = id;
>>>      > +     id = 0;
>>>      > +
>>>      >       seq_puts(m, "\tRelocated BOs:\n");
>>>      >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>      base.vm_status) {
>>>      >               if (!bo_va->base.bo <http://base.bo>)
>>>      > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>      amdgpu_vm *vm, struct seq_file *m)
>>>      >
>>>      >       seq_printf(m, "\tTotal idle size: %12lld\tobjs:\t%d\n",
>>>      total_idle,
>>>      >                  total_idle_objs);
>>>      > +     seq_printf(m, "\tTotal dirty size:  %12lld\tobjs:\t%d\n",
>>>      total_dirty,
>>>      > +                total_dirty_objs);
>>>      >       seq_printf(m, "\tTotal relocated size:
>>>       %12lld\tobjs:\t%d\n", total_relocated,
>>>      >                  total_relocated_objs);
>>>      >       seq_printf(m, "\tTotal moved size:  %12lld\tobjs:\t%d\n",
>>>      total_moved,
>>>      > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > index d9ab97eabda9..f91d4fcf80b8 100644
>>>      > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>      > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>>      >       /* per VM BOs moved, but not yet updated in the PT */
>>>      >       struct list_head        moved;
>>>      >
>>>      > +     /* normal and per VM BOs that are not moved, but have new
>>>      PT entries */
>>>      > +     struct list_head        dirty;
>>>      > +
>>>      >       /* All BOs of this VM not currently in the state machine */
>>>      >       struct list_head        idle;
>>>      >
>>>

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

[-- Attachment #1: Type: text/plain, Size: 14014 bytes --]


> On Oct 31, 2023, at 23:07, Christian König <christian.koenig@amd.com> wrote:
> 
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>> 
>> 
>> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com <mailto:christian.koenig@amd.com>> wrote:
>>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
>>> > BO specified in the GEM ioctl; however, when a binding is split, the
>>> > adjacent bindings also need to be updated. Such updates currently ends up
>>> > getting deferred until next submission which causes stalls.
>>> 
>>> Yeah, that is a necessity. The hardware simply doesn't support what you 
>>> try to do here in all cases.
>> 
>> What can the hardware not do here? Is this just needing to wait for TLB flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older than Polaris) you can't invalidate the TLB while it is in use.
> 
> For Polaris and older it just means that you don't have a guarantee that the shader can't access the memory any more. So delaying the free operation helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to invalidate the TLB while it is in use you can potentially triggering memory accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new VMID for each submission instead of invalidating the old one.

Thanks for the information. I looked into the VMID allocation logic and I see that if concurrent_flush is false, then we defer any flush (or VMID reuse that requires a flush) until that VMID is idle.

What patch #3 ends up doing is just performing the PT update right away. Any required TLB update is deferred by amdgpu_vm_update_range through the increment of tlb_seq, and amdgpu_vmid.c is responsible for doing the actual TLB flush in a manner that does not trigger the bug.

Can you confirm that this would be fine for the current hardware?

Tatsuyuki.

> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as well), but this is something you can really only do on some hw generations after validating that it works.
> 
> Regards,
> Christian. 
> 
>>  
>>> 
>>> So this approach won't work in general.
>>> 
>>> Regards,
>>> Christian.
>>> 
>>> >
>>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>>> > BOs, containing all BOs that have pending updates in `invalids`.
>>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
>>> > in the dirty state.
>>> >
>>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com <mailto:ishitatsuyuki@gmail.com>>
>>> > ---
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>>> >
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > index a1b15d0d6c48..01d3a97248b0 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>>> >    * vital here, so they are not reported back to userspace.
>>> >    */
>>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>> > -                                 struct amdgpu_vm *vm,
>>> > -                                 struct amdgpu_bo_va *bo_va,
>>> > -                                 uint32_t operation)
>>> > +                                 struct amdgpu_vm *vm)
>>> >   {
>>> > +     struct amdgpu_bo_va *bo_va;
>>> >       int r;
>>> >   
>>> >       if (!amdgpu_vm_ready(vm))
>>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>> >       if (r)
>>> >               goto error;
>>> >   
>>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>>> > +     spin_lock(&vm->status_lock);
>>> > +     while (!list_empty(&vm->dirty)) {
>>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>>> > +                                      base.vm_status);
>>> > +             spin_unlock(&vm->status_lock);
>>> > +
>>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>> >               if (r)
>>> >                       goto error;
>>> > +             spin_lock(&vm->status_lock);
>>> >       }
>>> > +     spin_unlock(&vm->status_lock);
>>> >   
>>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>> >   
>>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>> >               break;
>>> >       }
>>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
>>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>> > -                                     args->operation);
>>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>> >   
>>> >   error:
>>> >       drm_exec_fini(&exec);
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > index dd6f72e2a1d6..01d31891cd05 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>>> >       spin_unlock(&vm_bo->vm->status_lock);
>>> >   }
>>> >   
>>> > +/**
>>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>> > + *
>>> > + * @vm_bo: vm_bo which is dirty
>>> > + *
>>> > + * State for normal and per VM BOs that are not moved, but have new entries in
>>> > + * bo_va->invalids.
>>> > + */
>>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>> > +{
>>> > +     spin_lock(&vm_bo->vm->status_lock);
>>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>> > +     spin_unlock(&vm_bo->vm->status_lock);
>>> > +}
>>> > +
>>> >   /**
>>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>>> >    *
>>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>> >   
>>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>> > +
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>> >   
>>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>>> >                       dma_resv_unlock(resv);
>>> >               spin_lock(&vm->status_lock);
>>> >       }
>>> > +
>>> > +     while (!list_empty(&vm->dirty)) {
>>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>>> > +                                      base.vm_status);
>>> > +             spin_unlock(&vm->status_lock);
>>> > +
>>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>> > +             if (r)
>>> > +                     return r;
>>> > +             spin_lock(&vm->status_lock);
>>> > +     }
>>> >       spin_unlock(&vm->status_lock);
>>> >   
>>> >       return 0;
>>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>> >                                   struct amdgpu_bo_va_mapping *mapping)
>>> >   {
>>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>>> > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo/>;
>>> >   
>>> >       mapping->bo_va = bo_va;
>>> >       list_add(&mapping->list, &bo_va->invalids);
>>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>>> > +     if (!bo_va->base.moved)
>>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>>> >   
>>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>>> >               amdgpu_vm_prt_get(adev);
>>> >   
>>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -         !bo_va->base.moved) {
>>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>>> > -     }
>>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>> >   }
>>> >   
>>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >                       before->flags = tmp->flags;
>>> >                       before->bo_va = tmp->bo_va;
>>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>>> > +                     if (!tmp->bo_va->base.moved)
>>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>> >               }
>>> >   
>>> >               /* Remember mapping split at the end */
>>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >                       after->flags = tmp->flags;
>>> >                       after->bo_va = tmp->bo_va;
>>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>>> > +                     if (!tmp->bo_va->base.moved)
>>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>> >               }
>>> >   
>>> >               list_del(&tmp->list);
>>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >   
>>> >       /* Insert partial mapping before the range */
>>> >       if (!list_empty(&before->list)) {
>>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo <http://base.bo/>;
>>> > -
>>> >               amdgpu_vm_it_insert(before, &vm->va);
>>> >               if (before->flags & AMDGPU_PTE_PRT)
>>> >                       amdgpu_vm_prt_get(adev);
>>> > -
>>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -                 !before->bo_va->base.moved)
>>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>>> >       } else {
>>> >               kfree(before);
>>> >       }
>>> >   
>>> >       /* Insert partial mapping after the range */
>>> >       if (!list_empty(&after->list)) {
>>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo <http://base.bo/>;
>>> > -
>>> >               amdgpu_vm_it_insert(after, &vm->va);
>>> >               if (after->flags & AMDGPU_PTE_PRT)
>>> >                       amdgpu_vm_prt_get(adev);
>>> > -
>>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -                 !after->bo_va->base.moved)
>>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>>> >       } else {
>>> >               kfree(after);
>>> >       }
>>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>>> >       INIT_LIST_HEAD(&vm->evicted);
>>> >       INIT_LIST_HEAD(&vm->relocated);
>>> >       INIT_LIST_HEAD(&vm->moved);
>>> > +     INIT_LIST_HEAD(&vm->dirty);
>>> >       INIT_LIST_HEAD(&vm->idle);
>>> >       INIT_LIST_HEAD(&vm->invalidated);
>>> >       spin_lock_init(&vm->status_lock);
>>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >   {
>>> >       struct amdgpu_bo_va *bo_va, *tmp;
>>> >       u64 total_idle = 0;
>>> > +     u64 total_dirty = 0;
>>> >       u64 total_relocated = 0;
>>> >       u64 total_moved = 0;
>>> >       u64 total_invalidated = 0;
>>> >       u64 total_done = 0;
>>> >       unsigned int total_idle_objs = 0;
>>> > +     unsigned int total_dirty_objs = 0;
>>> >       unsigned int total_relocated_objs = 0;
>>> >       unsigned int total_moved_objs = 0;
>>> >       unsigned int total_invalidated_objs = 0;
>>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >       total_idle_objs = id;
>>> >       id = 0;
>>> >   
>>> > +     seq_puts(m, "\tDirty BOs:\n");
>>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>>> > +             if (!bo_va->base.bo <http://base.bo/>)
>>> > +                     continue;
>>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo <http://base.bo/>, m);
>>> > +     }
>>> > +     total_dirty_objs = id;
>>> > +     id = 0;
>>> > +
>>> >       seq_puts(m, "\tRelocated BOs:\n");
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>> >               if (!bo_va->base.bo <http://base.bo/>)
>>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >   
>>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>> >                  total_idle_objs);
>>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
>>> > +                total_dirty_objs);
>>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>> >                  total_relocated_objs);
>>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > index d9ab97eabda9..f91d4fcf80b8 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>> >       /* per VM BOs moved, but not yet updated in the PT */
>>> >       struct list_head        moved;
>>> >   
>>> > +     /* normal and per VM BOs that are not moved, but have new PT entries */
>>> > +     struct list_head        dirty;
>>> > +
>>> >       /* All BOs of this VM not currently in the state machine */
>>> >       struct list_head        idle;
>>> >   
>>> 
> 


[-- Attachment #2: Type: text/html, Size: 27243 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Tatsuyuki Ishi]

[-- Attachment #1: Type: text/plain, Size: 14014 bytes --]


> On Oct 31, 2023, at 23:07, Christian König <christian.koenig@amd.com> wrote:
> 
> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>> 
>> 
>> On Tue, Oct 31, 2023 at 2:57 PM Christian König <christian.koenig@amd.com <mailto:christian.koenig@amd.com>> wrote:
>>> Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>> > The current amdgpu_gem_va_update_vm only tries to perform updates for the
>>> > BO specified in the GEM ioctl; however, when a binding is split, the
>>> > adjacent bindings also need to be updated. Such updates currently ends up
>>> > getting deferred until next submission which causes stalls.
>>> 
>>> Yeah, that is a necessity. The hardware simply doesn't support what you 
>>> try to do here in all cases.
>> 
>> What can the hardware not do here? Is this just needing to wait for TLB flushes before we can free pagetables, can we just delay that?
> 
> On some hardware generations (especially Navi1x, but also everything older than Polaris) you can't invalidate the TLB while it is in use.
> 
> For Polaris and older it just means that you don't have a guarantee that the shader can't access the memory any more. So delaying the free operation helps here.
> 
> But for Navi1x it's a workaround for a hardware bug. If you try to invalidate the TLB while it is in use you can potentially triggering memory accesses to random addresses.
> 
> That's why we still delay TLB invalidation's to the next CS and use a new VMID for each submission instead of invalidating the old one.

Thanks for the information. I looked into the VMID allocation logic and I see that if concurrent_flush is false, then we defer any flush (or VMID reuse that requires a flush) until that VMID is idle.

What patch #3 ends up doing is just performing the PT update right away. Any required TLB update is deferred by amdgpu_vm_update_range through the increment of tlb_seq, and amdgpu_vmid.c is responsible for doing the actual TLB flush in a manner that does not trigger the bug.

Can you confirm that this would be fine for the current hardware?

Tatsuyuki.

> 
> I'm currently working on changing that for Navi2x and newer (maybe Vega as well), but this is something you can really only do on some hw generations after validating that it works.
> 
> Regards,
> Christian. 
> 
>>  
>>> 
>>> So this approach won't work in general.
>>> 
>>> Regards,
>>> Christian.
>>> 
>>> >
>>> > Introduce a new state "dirty", shared between per-VM BOs and traditional
>>> > BOs, containing all BOs that have pending updates in `invalids`.
>>> > amdgpu_gem_va_update_vm will now simply flush any pending updates for BOs
>>> > in the dirty state.
>>> >
>>> > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com <mailto:ishitatsuyuki@gmail.com>>
>>> > ---
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66 ++++++++++++++++++-------
>>> >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |  3 ++
>>> >   3 files changed, 63 insertions(+), 24 deletions(-)
>>> >
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > index a1b15d0d6c48..01d3a97248b0 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct drm_device *dev, void *data,
>>> >    * vital here, so they are not reported back to userspace.
>>> >    */
>>> >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>> > -                                 struct amdgpu_vm *vm,
>>> > -                                 struct amdgpu_bo_va *bo_va,
>>> > -                                 uint32_t operation)
>>> > +                                 struct amdgpu_vm *vm)
>>> >   {
>>> > +     struct amdgpu_bo_va *bo_va;
>>> >       int r;
>>> >   
>>> >       if (!amdgpu_vm_ready(vm))
>>> > @@ -617,12 +616,18 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>> >       if (r)
>>> >               goto error;
>>> >   
>>> > -     if (operation == AMDGPU_VA_OP_MAP ||
>>> > -         operation == AMDGPU_VA_OP_REPLACE) {
>>> > +     spin_lock(&vm->status_lock);
>>> > +     while (!list_empty(&vm->dirty)) {
>>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>>> > +                                      base.vm_status);
>>> > +             spin_unlock(&vm->status_lock);
>>> > +
>>> >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>> >               if (r)
>>> >                       goto error;
>>> > +             spin_lock(&vm->status_lock);
>>> >       }
>>> > +     spin_unlock(&vm->status_lock);
>>> >   
>>> >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>> >   
>>> > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>> >               break;
>>> >       }
>>> >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !amdgpu_vm_debug)
>>> > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>> > -                                     args->operation);
>>> > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>> >   
>>> >   error:
>>> >       drm_exec_fini(&exec);
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > index dd6f72e2a1d6..01d31891cd05 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> > @@ -191,6 +191,21 @@ static void amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>>> >       spin_unlock(&vm_bo->vm->status_lock);
>>> >   }
>>> >   
>>> > +/**
>>> > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>> > + *
>>> > + * @vm_bo: vm_bo which is dirty
>>> > + *
>>> > + * State for normal and per VM BOs that are not moved, but have new entries in
>>> > + * bo_va->invalids.
>>> > + */
>>> > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>> > +{
>>> > +     spin_lock(&vm_bo->vm->status_lock);
>>> > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>> > +     spin_unlock(&vm_bo->vm->status_lock);
>>> > +}
>>> > +
>>> >   /**
>>> >    * amdgpu_vm_bo_moved - vm_bo is moved
>>> >    *
>>> > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct amdgpu_vm *vm,
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted, base.eviction_status)
>>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>> >   
>>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status)
>>> > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>> > +
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status)
>>> >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>> >   
>>> > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
>>> >                       dma_resv_unlock(resv);
>>> >               spin_lock(&vm->status_lock);
>>> >       }
>>> > +
>>> > +     while (!list_empty(&vm->dirty)) {
>>> > +             bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
>>> > +                                      base.vm_status);
>>> > +             spin_unlock(&vm->status_lock);
>>> > +
>>> > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>> > +             if (r)
>>> > +                     return r;
>>> > +             spin_lock(&vm->status_lock);
>>> > +     }
>>> >       spin_unlock(&vm->status_lock);
>>> >   
>>> >       return 0;
>>> > @@ -1476,19 +1505,16 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>> >                                   struct amdgpu_bo_va_mapping *mapping)
>>> >   {
>>> >       struct amdgpu_vm *vm = bo_va->base.vm;
>>> > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo/>;
>>> >   
>>> >       mapping->bo_va = bo_va;
>>> >       list_add(&mapping->list, &bo_va->invalids);
>>> >       amdgpu_vm_it_insert(mapping, &vm->va);
>>> > +     if (!bo_va->base.moved)
>>> > +             amdgpu_vm_bo_dirty(&bo_va->base);
>>> >   
>>> >       if (mapping->flags & AMDGPU_PTE_PRT)
>>> >               amdgpu_vm_prt_get(adev);
>>> >   
>>> > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -         !bo_va->base.moved) {
>>> > -             amdgpu_vm_bo_moved(&bo_va->base);
>>> > -     }
>>> >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>> >   }
>>> >   
>>> > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >                       before->flags = tmp->flags;
>>> >                       before->bo_va = tmp->bo_va;
>>> >                       list_add(&before->list, &tmp->bo_va->invalids);
>>> > +                     if (!tmp->bo_va->base.moved)
>>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>> >               }
>>> >   
>>> >               /* Remember mapping split at the end */
>>> > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >                       after->flags = tmp->flags;
>>> >                       after->bo_va = tmp->bo_va;
>>> >                       list_add(&after->list, &tmp->bo_va->invalids);
>>> > +                     if (!tmp->bo_va->base.moved)
>>> > +                             amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>> >               }
>>> >   
>>> >               list_del(&tmp->list);
>>> > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> >   
>>> >       /* Insert partial mapping before the range */
>>> >       if (!list_empty(&before->list)) {
>>> > -             struct amdgpu_bo *bo = before->bo_va->base.bo <http://base.bo/>;
>>> > -
>>> >               amdgpu_vm_it_insert(before, &vm->va);
>>> >               if (before->flags & AMDGPU_PTE_PRT)
>>> >                       amdgpu_vm_prt_get(adev);
>>> > -
>>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -                 !before->bo_va->base.moved)
>>> > -                     amdgpu_vm_bo_moved(&before->bo_va->base);
>>> >       } else {
>>> >               kfree(before);
>>> >       }
>>> >   
>>> >       /* Insert partial mapping after the range */
>>> >       if (!list_empty(&after->list)) {
>>> > -             struct amdgpu_bo *bo = after->bo_va->base.bo <http://base.bo/>;
>>> > -
>>> >               amdgpu_vm_it_insert(after, &vm->va);
>>> >               if (after->flags & AMDGPU_PTE_PRT)
>>> >                       amdgpu_vm_prt_get(adev);
>>> > -
>>> > -             if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>> > -                 !after->bo_va->base.moved)
>>> > -                     amdgpu_vm_bo_moved(&after->bo_va->base);
>>> >       } else {
>>> >               kfree(after);
>>> >       }
>>> > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp
>>> >       INIT_LIST_HEAD(&vm->evicted);
>>> >       INIT_LIST_HEAD(&vm->relocated);
>>> >       INIT_LIST_HEAD(&vm->moved);
>>> > +     INIT_LIST_HEAD(&vm->dirty);
>>> >       INIT_LIST_HEAD(&vm->idle);
>>> >       INIT_LIST_HEAD(&vm->invalidated);
>>> >       spin_lock_init(&vm->status_lock);
>>> > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >   {
>>> >       struct amdgpu_bo_va *bo_va, *tmp;
>>> >       u64 total_idle = 0;
>>> > +     u64 total_dirty = 0;
>>> >       u64 total_relocated = 0;
>>> >       u64 total_moved = 0;
>>> >       u64 total_invalidated = 0;
>>> >       u64 total_done = 0;
>>> >       unsigned int total_idle_objs = 0;
>>> > +     unsigned int total_dirty_objs = 0;
>>> >       unsigned int total_relocated_objs = 0;
>>> >       unsigned int total_moved_objs = 0;
>>> >       unsigned int total_invalidated_objs = 0;
>>> > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >       total_idle_objs = id;
>>> >       id = 0;
>>> >   
>>> > +     seq_puts(m, "\tDirty BOs:\n");
>>> > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty, base.vm_status) {
>>> > +             if (!bo_va->base.bo <http://base.bo/>)
>>> > +                     continue;
>>> > +             total_dirty += amdgpu_bo_print_info(id++, bo_va->base.bo <http://base.bo/>, m);
>>> > +     }
>>> > +     total_dirty_objs = id;
>>> > +     id = 0;
>>> > +
>>> >       seq_puts(m, "\tRelocated BOs:\n");
>>> >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated, base.vm_status) {
>>> >               if (!bo_va->base.bo <http://base.bo/>)
>>> > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct amdgpu_vm *vm, struct seq_file *m)
>>> >   
>>> >       seq_printf(m, "\tTotal idle size:        %12lld\tobjs:\t%d\n", total_idle,
>>> >                  total_idle_objs);
>>> > +     seq_printf(m, "\tTotal dirty size:       %12lld\tobjs:\t%d\n", total_dirty,
>>> > +                total_dirty_objs);
>>> >       seq_printf(m, "\tTotal relocated size:   %12lld\tobjs:\t%d\n", total_relocated,
>>> >                  total_relocated_objs);
>>> >       seq_printf(m, "\tTotal moved size:       %12lld\tobjs:\t%d\n", total_moved,
>>> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > index d9ab97eabda9..f91d4fcf80b8 100644
>>> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>> >       /* per VM BOs moved, but not yet updated in the PT */
>>> >       struct list_head        moved;
>>> >   
>>> > +     /* normal and per VM BOs that are not moved, but have new PT entries */
>>> > +     struct list_head        dirty;
>>> > +
>>> >       /* All BOs of this VM not currently in the state machine */
>>> >       struct list_head        idle;
>>> >   
>>> 
> 


[-- Attachment #2: Type: text/html, Size: 27243 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 15681 bytes --]

Am 06.11.23 um 08:56 schrieb Tatsuyuki Ishi:
>
>> On Oct 31, 2023, at 23:07, Christian König <christian.koenig@amd.com> 
>> wrote:
>>
>> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>>>
>>>
>>> On Tue, Oct 31, 2023 at 2:57 PM Christian König 
>>> <christian.koenig@amd.com> wrote:
>>>
>>>     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>>     > The current amdgpu_gem_va_update_vm only tries to perform
>>>     updates for the
>>>     > BO specified in the GEM ioctl; however, when a binding is
>>>     split, the
>>>     > adjacent bindings also need to be updated. Such updates
>>>     currently ends up
>>>     > getting deferred until next submission which causes stalls.
>>>
>>>     Yeah, that is a necessity. The hardware simply doesn't support
>>>     what you
>>>     try to do here in all cases.
>>>
>>>
>>> What can the hardware not do here? Is this just needing to wait for 
>>> TLB flushes before we can free pagetables, can we just delay that?
>>
>> On some hardware generations (especially Navi1x, but also everything 
>> older than Polaris) you can't invalidate the TLB while it is in use.
>>
>> For Polaris and older it just means that you don't have a guarantee 
>> that the shader can't access the memory any more. So delaying the 
>> free operation helps here.
>>
>> But for Navi1x it's a workaround for a hardware bug. If you try to 
>> invalidate the TLB while it is in use you can potentially triggering 
>> memory accesses to random addresses.
>>
>> That's why we still delay TLB invalidation's to the next CS and use a 
>> new VMID for each submission instead of invalidating the old one.
>
> Thanks for the information. I looked into the VMID allocation logic 
> and I see that if concurrent_flush is false, then we defer any flush 
> (or VMID reuse that requires a flush) until that VMID is idle.
>
> What patch #3 ends up doing is just performing the PT update right 
> away. Any required TLB update is deferred by amdgpu_vm_update_range 
> through the increment of tlb_seq, and amdgpu_vmid.c is responsible for 
> doing the actual TLB flush in a manner that does not trigger the bug.
>
> Can you confirm that this would be fine for the current hardware?

Yeah, that should work. I'm just think about the UAPI a bit, we should 
probably improve this to use something like a drm_syncobj instead of 
just a flag to be future prove.

Christian.

>
> Tatsuyuki.
>
>>
>> I'm currently working on changing that for Navi2x and newer (maybe 
>> Vega as well), but this is something you can really only do on some 
>> hw generations after validating that it works.
>>
>> Regards,
>> Christian.
>>
>>>
>>>
>>>     So this approach won't work in general.
>>>
>>>     Regards,
>>>     Christian.
>>>
>>>     >
>>>     > Introduce a new state "dirty", shared between per-VM BOs and
>>>     traditional
>>>     > BOs, containing all BOs that have pending updates in `invalids`.
>>>     > amdgpu_gem_va_update_vm will now simply flush any pending
>>>     updates for BOs
>>>     > in the dirty state.
>>>     >
>>>     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>>     > ---
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>>>     ++++++++++++++++++-------
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  | 3 ++
>>>     >   3 files changed, 63 insertions(+), 24 deletions(-)
>>>     >
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > index a1b15d0d6c48..01d3a97248b0 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>>>     drm_device *dev, void *data,
>>>     >    * vital here, so they are not reported back to userspace.
>>>     >    */
>>>     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>     > -                                 struct amdgpu_vm *vm,
>>>     > -                                 struct amdgpu_bo_va *bo_va,
>>>     > -                                 uint32_t operation)
>>>     > +                                 struct amdgpu_vm *vm)
>>>     >   {
>>>     > +     struct amdgpu_bo_va *bo_va;
>>>     >       int r;
>>>     >
>>>     >       if (!amdgpu_vm_ready(vm))
>>>     > @@ -617,12 +616,18 @@ static void
>>>     amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>     >       if (r)
>>>     >               goto error;
>>>     >
>>>     > -     if (operation == AMDGPU_VA_OP_MAP ||
>>>     > -         operation == AMDGPU_VA_OP_REPLACE) {
>>>     > +     spin_lock(&vm->status_lock);
>>>     > +     while (!list_empty(&vm->dirty)) {
>>>     > +             bo_va = list_first_entry(&vm->dirty, struct
>>>     amdgpu_bo_va,
>>>     > + base.vm_status);
>>>     > +  spin_unlock(&vm->status_lock);
>>>     > +
>>>     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>     >               if (r)
>>>     >                       goto error;
>>>     > +  spin_lock(&vm->status_lock);
>>>     >       }
>>>     > +     spin_unlock(&vm->status_lock);
>>>     >
>>>     >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>>     >
>>>     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>>>     *dev, void *data,
>>>     >               break;
>>>     >       }
>>>     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>>>     !amdgpu_vm_debug)
>>>     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>>     > -  args->operation);
>>>     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>>     >
>>>     >   error:
>>>     >       drm_exec_fini(&exec);
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > index dd6f72e2a1d6..01d31891cd05 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > @@ -191,6 +191,21 @@ static void
>>>     amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>>>     >  spin_unlock(&vm_bo->vm->status_lock);
>>>     >   }
>>>     >
>>>     > +/**
>>>     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>>     > + *
>>>     > + * @vm_bo: vm_bo which is dirty
>>>     > + *
>>>     > + * State for normal and per VM BOs that are not moved, but
>>>     have new entries in
>>>     > + * bo_va->invalids.
>>>     > + */
>>>     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>>     > +{
>>>     > +  spin_lock(&vm_bo->vm->status_lock);
>>>     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>>     > +  spin_unlock(&vm_bo->vm->status_lock);
>>>     > +}
>>>     > +
>>>     >   /**
>>>     >    * amdgpu_vm_bo_moved - vm_bo is moved
>>>     >    *
>>>     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct
>>>     amdgpu_vm *vm,
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>>>     base.eviction_status)
>>>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     >
>>>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>     base.vm_status)
>>>     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     > +
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>     base.vm_status)
>>>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     >
>>>     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>>>     amdgpu_device *adev,
>>>     >                       dma_resv_unlock(resv);
>>>     >  spin_lock(&vm->status_lock);
>>>     >       }
>>>     > +
>>>     > +     while (!list_empty(&vm->dirty)) {
>>>     > +             bo_va = list_first_entry(&vm->dirty, struct
>>>     amdgpu_bo_va,
>>>     > + base.vm_status);
>>>     > +  spin_unlock(&vm->status_lock);
>>>     > +
>>>     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>     > +             if (r)
>>>     > +                     return r;
>>>     > +  spin_lock(&vm->status_lock);
>>>     > +     }
>>>     >       spin_unlock(&vm->status_lock);
>>>     >
>>>     >       return 0;
>>>     > @@ -1476,19 +1505,16 @@ static void
>>>     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>>     >                                   struct amdgpu_bo_va_mapping
>>>     *mapping)
>>>     >   {
>>>     >       struct amdgpu_vm *vm = bo_va->base.vm;
>>>     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo/>;
>>>     >
>>>     >       mapping->bo_va = bo_va;
>>>     >       list_add(&mapping->list, &bo_va->invalids);
>>>     >       amdgpu_vm_it_insert(mapping, &vm->va);
>>>     > +     if (!bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&bo_va->base);
>>>     >
>>>     >       if (mapping->flags & AMDGPU_PTE_PRT)
>>>     >               amdgpu_vm_prt_get(adev);
>>>     >
>>>     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>>     > -         !bo_va->base.moved) {
>>>     > -  amdgpu_vm_bo_moved(&bo_va->base);
>>>     > -     }
>>>     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>>     >   }
>>>     >
>>>     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >                       before->flags = tmp->flags;
>>>     >                       before->bo_va = tmp->bo_va;
>>>     >  list_add(&before->list, &tmp->bo_va->invalids);
>>>     > +                     if (!tmp->bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>     >               }
>>>     >
>>>     >               /* Remember mapping split at the end */
>>>     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >                       after->flags = tmp->flags;
>>>     >                       after->bo_va = tmp->bo_va;
>>>     >  list_add(&after->list, &tmp->bo_va->invalids);
>>>     > +                     if (!tmp->bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>     >               }
>>>     >
>>>     >               list_del(&tmp->list);
>>>     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >
>>>     >       /* Insert partial mapping before the range */
>>>     >       if (!list_empty(&before->list)) {
>>>     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>>>     <http://base.bo/>;
>>>     > -
>>>     >               amdgpu_vm_it_insert(before, &vm->va);
>>>     >               if (before->flags & AMDGPU_PTE_PRT)
>>>     >  amdgpu_vm_prt_get(adev);
>>>     > -
>>>     > -             if (bo && bo->tbo.base.resv ==
>>>     vm->root.bo->tbo.base.resv &&
>>>     > -  !before->bo_va->base.moved)
>>>     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>>>     >       } else {
>>>     >               kfree(before);
>>>     >       }
>>>     >
>>>     >       /* Insert partial mapping after the range */
>>>     >       if (!list_empty(&after->list)) {
>>>     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>>>     <http://base.bo/>;
>>>     > -
>>>     >               amdgpu_vm_it_insert(after, &vm->va);
>>>     >               if (after->flags & AMDGPU_PTE_PRT)
>>>     >  amdgpu_vm_prt_get(adev);
>>>     > -
>>>     > -             if (bo && bo->tbo.base.resv ==
>>>     vm->root.bo->tbo.base.resv &&
>>>     > -  !after->bo_va->base.moved)
>>>     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>>>     >       } else {
>>>     >               kfree(after);
>>>     >       }
>>>     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>>>     *adev, struct amdgpu_vm *vm, int32_t xcp
>>>     >       INIT_LIST_HEAD(&vm->evicted);
>>>     >       INIT_LIST_HEAD(&vm->relocated);
>>>     >       INIT_LIST_HEAD(&vm->moved);
>>>     > +     INIT_LIST_HEAD(&vm->dirty);
>>>     >       INIT_LIST_HEAD(&vm->idle);
>>>     >  INIT_LIST_HEAD(&vm->invalidated);
>>>     >  spin_lock_init(&vm->status_lock);
>>>     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >   {
>>>     >       struct amdgpu_bo_va *bo_va, *tmp;
>>>     >       u64 total_idle = 0;
>>>     > +     u64 total_dirty = 0;
>>>     >       u64 total_relocated = 0;
>>>     >       u64 total_moved = 0;
>>>     >       u64 total_invalidated = 0;
>>>     >       u64 total_done = 0;
>>>     >       unsigned int total_idle_objs = 0;
>>>     > +     unsigned int total_dirty_objs = 0;
>>>     >       unsigned int total_relocated_objs = 0;
>>>     >       unsigned int total_moved_objs = 0;
>>>     >       unsigned int total_invalidated_objs = 0;
>>>     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >       total_idle_objs = id;
>>>     >       id = 0;
>>>     >
>>>     > +     seq_puts(m, "\tDirty BOs:\n");
>>>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>     base.vm_status) {
>>>     > +             if (!bo_va->base.bo <http://base.bo/>)
>>>     > +                     continue;
>>>     > +             total_dirty += amdgpu_bo_print_info(id++,
>>>     bo_va->base.bo <http://base.bo/>, m);
>>>     > +     }
>>>     > +     total_dirty_objs = id;
>>>     > +     id = 0;
>>>     > +
>>>     >       seq_puts(m, "\tRelocated BOs:\n");
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>     base.vm_status) {
>>>     >               if (!bo_va->base.bo <http://base.bo/>)
>>>     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >
>>>     >       seq_printf(m, "\tTotal idle size:  
>>>     %12lld\tobjs:\t%d\n", total_idle,
>>>     >                  total_idle_objs);
>>>     > +     seq_printf(m, "\tTotal dirty size:
>>>      %12lld\tobjs:\t%d\n", total_dirty,
>>>     > +                total_dirty_objs);
>>>     >       seq_printf(m, "\tTotal relocated size:
>>>      %12lld\tobjs:\t%d\n", total_relocated,
>>>     >                  total_relocated_objs);
>>>     >       seq_printf(m, "\tTotal moved size:
>>>      %12lld\tobjs:\t%d\n", total_moved,
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > index d9ab97eabda9..f91d4fcf80b8 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>>     >       /* per VM BOs moved, but not yet updated in the PT */
>>>     >       struct list_head        moved;
>>>     >
>>>     > +     /* normal and per VM BOs that are not moved, but have
>>>     new PT entries */
>>>     > +     struct list_head        dirty;
>>>     > +
>>>     >       /* All BOs of this VM not currently in the state machine */
>>>     >       struct list_head        idle;
>>>     >
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 33080 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 15681 bytes --]

Am 06.11.23 um 08:56 schrieb Tatsuyuki Ishi:
>
>> On Oct 31, 2023, at 23:07, Christian König <christian.koenig@amd.com> 
>> wrote:
>>
>> Am 31.10.23 um 14:59 schrieb Bas Nieuwenhuizen:
>>>
>>>
>>> On Tue, Oct 31, 2023 at 2:57 PM Christian König 
>>> <christian.koenig@amd.com> wrote:
>>>
>>>     Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
>>>     > The current amdgpu_gem_va_update_vm only tries to perform
>>>     updates for the
>>>     > BO specified in the GEM ioctl; however, when a binding is
>>>     split, the
>>>     > adjacent bindings also need to be updated. Such updates
>>>     currently ends up
>>>     > getting deferred until next submission which causes stalls.
>>>
>>>     Yeah, that is a necessity. The hardware simply doesn't support
>>>     what you
>>>     try to do here in all cases.
>>>
>>>
>>> What can the hardware not do here? Is this just needing to wait for 
>>> TLB flushes before we can free pagetables, can we just delay that?
>>
>> On some hardware generations (especially Navi1x, but also everything 
>> older than Polaris) you can't invalidate the TLB while it is in use.
>>
>> For Polaris and older it just means that you don't have a guarantee 
>> that the shader can't access the memory any more. So delaying the 
>> free operation helps here.
>>
>> But for Navi1x it's a workaround for a hardware bug. If you try to 
>> invalidate the TLB while it is in use you can potentially triggering 
>> memory accesses to random addresses.
>>
>> That's why we still delay TLB invalidation's to the next CS and use a 
>> new VMID for each submission instead of invalidating the old one.
>
> Thanks for the information. I looked into the VMID allocation logic 
> and I see that if concurrent_flush is false, then we defer any flush 
> (or VMID reuse that requires a flush) until that VMID is idle.
>
> What patch #3 ends up doing is just performing the PT update right 
> away. Any required TLB update is deferred by amdgpu_vm_update_range 
> through the increment of tlb_seq, and amdgpu_vmid.c is responsible for 
> doing the actual TLB flush in a manner that does not trigger the bug.
>
> Can you confirm that this would be fine for the current hardware?

Yeah, that should work. I'm just think about the UAPI a bit, we should 
probably improve this to use something like a drm_syncobj instead of 
just a flag to be future prove.

Christian.

>
> Tatsuyuki.
>
>>
>> I'm currently working on changing that for Navi2x and newer (maybe 
>> Vega as well), but this is something you can really only do on some 
>> hw generations after validating that it works.
>>
>> Regards,
>> Christian.
>>
>>>
>>>
>>>     So this approach won't work in general.
>>>
>>>     Regards,
>>>     Christian.
>>>
>>>     >
>>>     > Introduce a new state "dirty", shared between per-VM BOs and
>>>     traditional
>>>     > BOs, containing all BOs that have pending updates in `invalids`.
>>>     > amdgpu_gem_va_update_vm will now simply flush any pending
>>>     updates for BOs
>>>     > in the dirty state.
>>>     >
>>>     > Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>>     > ---
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 18 ++++---
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 66
>>>     ++++++++++++++++++-------
>>>     >   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  | 3 ++
>>>     >   3 files changed, 63 insertions(+), 24 deletions(-)
>>>     >
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > index a1b15d0d6c48..01d3a97248b0 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>>     > @@ -604,10 +604,9 @@ int amdgpu_gem_metadata_ioctl(struct
>>>     drm_device *dev, void *data,
>>>     >    * vital here, so they are not reported back to userspace.
>>>     >    */
>>>     >   static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>     > -                                 struct amdgpu_vm *vm,
>>>     > -                                 struct amdgpu_bo_va *bo_va,
>>>     > -                                 uint32_t operation)
>>>     > +                                 struct amdgpu_vm *vm)
>>>     >   {
>>>     > +     struct amdgpu_bo_va *bo_va;
>>>     >       int r;
>>>     >
>>>     >       if (!amdgpu_vm_ready(vm))
>>>     > @@ -617,12 +616,18 @@ static void
>>>     amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
>>>     >       if (r)
>>>     >               goto error;
>>>     >
>>>     > -     if (operation == AMDGPU_VA_OP_MAP ||
>>>     > -         operation == AMDGPU_VA_OP_REPLACE) {
>>>     > +     spin_lock(&vm->status_lock);
>>>     > +     while (!list_empty(&vm->dirty)) {
>>>     > +             bo_va = list_first_entry(&vm->dirty, struct
>>>     amdgpu_bo_va,
>>>     > + base.vm_status);
>>>     > +  spin_unlock(&vm->status_lock);
>>>     > +
>>>     >               r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>     >               if (r)
>>>     >                       goto error;
>>>     > +  spin_lock(&vm->status_lock);
>>>     >       }
>>>     > +     spin_unlock(&vm->status_lock);
>>>     >
>>>     >       r = amdgpu_vm_update_pdes(adev, vm, false);
>>>     >
>>>     > @@ -792,8 +797,7 @@ int amdgpu_gem_va_ioctl(struct drm_device
>>>     *dev, void *data,
>>>     >               break;
>>>     >       }
>>>     >       if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) &&
>>>     !amdgpu_vm_debug)
>>>     > -             amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
>>>     > -  args->operation);
>>>     > +             amdgpu_gem_va_update_vm(adev, &fpriv->vm);
>>>     >
>>>     >   error:
>>>     >       drm_exec_fini(&exec);
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > index dd6f72e2a1d6..01d31891cd05 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>     > @@ -191,6 +191,21 @@ static void
>>>     amdgpu_vm_bo_set_evicted(struct amdgpu_vm_bo_base *vm_bo, bool evict
>>>     >  spin_unlock(&vm_bo->vm->status_lock);
>>>     >   }
>>>     >
>>>     > +/**
>>>     > + * amdgpu_vm_bo_dirty - vm_bo is dirty
>>>     > + *
>>>     > + * @vm_bo: vm_bo which is dirty
>>>     > + *
>>>     > + * State for normal and per VM BOs that are not moved, but
>>>     have new entries in
>>>     > + * bo_va->invalids.
>>>     > + */
>>>     > +static void amdgpu_vm_bo_dirty(struct amdgpu_vm_bo_base *vm_bo)
>>>     > +{
>>>     > +  spin_lock(&vm_bo->vm->status_lock);
>>>     > +     list_move(&vm_bo->vm_status, &vm_bo->vm->dirty);
>>>     > +  spin_unlock(&vm_bo->vm->status_lock);
>>>     > +}
>>>     > +
>>>     >   /**
>>>     >    * amdgpu_vm_bo_moved - vm_bo is moved
>>>     >    *
>>>     > @@ -1042,6 +1057,9 @@ void amdgpu_vm_get_memory(struct
>>>     amdgpu_vm *vm,
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->evicted,
>>>     base.eviction_status)
>>>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     >
>>>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>     base.vm_status)
>>>     > +             amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     > +
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>     base.vm_status)
>>>     >               amdgpu_vm_bo_get_memory(bo_va, stats);
>>>     >
>>>     > @@ -1411,6 +1429,17 @@ int amdgpu_vm_handle_moved(struct
>>>     amdgpu_device *adev,
>>>     >                       dma_resv_unlock(resv);
>>>     >  spin_lock(&vm->status_lock);
>>>     >       }
>>>     > +
>>>     > +     while (!list_empty(&vm->dirty)) {
>>>     > +             bo_va = list_first_entry(&vm->dirty, struct
>>>     amdgpu_bo_va,
>>>     > + base.vm_status);
>>>     > +  spin_unlock(&vm->status_lock);
>>>     > +
>>>     > +             r = amdgpu_vm_bo_update(adev, bo_va, false);
>>>     > +             if (r)
>>>     > +                     return r;
>>>     > +  spin_lock(&vm->status_lock);
>>>     > +     }
>>>     >       spin_unlock(&vm->status_lock);
>>>     >
>>>     >       return 0;
>>>     > @@ -1476,19 +1505,16 @@ static void
>>>     amdgpu_vm_bo_insert_map(struct amdgpu_device *adev,
>>>     >                                   struct amdgpu_bo_va_mapping
>>>     *mapping)
>>>     >   {
>>>     >       struct amdgpu_vm *vm = bo_va->base.vm;
>>>     > -     struct amdgpu_bo *bo = bo_va->base.bo <http://base.bo/>;
>>>     >
>>>     >       mapping->bo_va = bo_va;
>>>     >       list_add(&mapping->list, &bo_va->invalids);
>>>     >       amdgpu_vm_it_insert(mapping, &vm->va);
>>>     > +     if (!bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&bo_va->base);
>>>     >
>>>     >       if (mapping->flags & AMDGPU_PTE_PRT)
>>>     >               amdgpu_vm_prt_get(adev);
>>>     >
>>>     > -     if (bo && bo->tbo.base.resv == vm->root.bo->tbo.base.resv &&
>>>     > -         !bo_va->base.moved) {
>>>     > -  amdgpu_vm_bo_moved(&bo_va->base);
>>>     > -     }
>>>     >       trace_amdgpu_vm_bo_map(bo_va, mapping);
>>>     >   }
>>>     >
>>>     > @@ -1725,6 +1751,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >                       before->flags = tmp->flags;
>>>     >                       before->bo_va = tmp->bo_va;
>>>     >  list_add(&before->list, &tmp->bo_va->invalids);
>>>     > +                     if (!tmp->bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>     >               }
>>>     >
>>>     >               /* Remember mapping split at the end */
>>>     > @@ -1736,6 +1764,8 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >                       after->flags = tmp->flags;
>>>     >                       after->bo_va = tmp->bo_va;
>>>     >  list_add(&after->list, &tmp->bo_va->invalids);
>>>     > +                     if (!tmp->bo_va->base.moved)
>>>     > +  amdgpu_vm_bo_dirty(&tmp->bo_va->base);
>>>     >               }
>>>     >
>>>     >               list_del(&tmp->list);
>>>     > @@ -1761,30 +1791,18 @@ int amdgpu_vm_bo_clear_mappings(struct
>>>     amdgpu_device *adev,
>>>     >
>>>     >       /* Insert partial mapping before the range */
>>>     >       if (!list_empty(&before->list)) {
>>>     > -             struct amdgpu_bo *bo = before->bo_va->base.bo
>>>     <http://base.bo/>;
>>>     > -
>>>     >               amdgpu_vm_it_insert(before, &vm->va);
>>>     >               if (before->flags & AMDGPU_PTE_PRT)
>>>     >  amdgpu_vm_prt_get(adev);
>>>     > -
>>>     > -             if (bo && bo->tbo.base.resv ==
>>>     vm->root.bo->tbo.base.resv &&
>>>     > -  !before->bo_va->base.moved)
>>>     > -  amdgpu_vm_bo_moved(&before->bo_va->base);
>>>     >       } else {
>>>     >               kfree(before);
>>>     >       }
>>>     >
>>>     >       /* Insert partial mapping after the range */
>>>     >       if (!list_empty(&after->list)) {
>>>     > -             struct amdgpu_bo *bo = after->bo_va->base.bo
>>>     <http://base.bo/>;
>>>     > -
>>>     >               amdgpu_vm_it_insert(after, &vm->va);
>>>     >               if (after->flags & AMDGPU_PTE_PRT)
>>>     >  amdgpu_vm_prt_get(adev);
>>>     > -
>>>     > -             if (bo && bo->tbo.base.resv ==
>>>     vm->root.bo->tbo.base.resv &&
>>>     > -  !after->bo_va->base.moved)
>>>     > -  amdgpu_vm_bo_moved(&after->bo_va->base);
>>>     >       } else {
>>>     >               kfree(after);
>>>     >       }
>>>     > @@ -2136,6 +2154,7 @@ int amdgpu_vm_init(struct amdgpu_device
>>>     *adev, struct amdgpu_vm *vm, int32_t xcp
>>>     >       INIT_LIST_HEAD(&vm->evicted);
>>>     >       INIT_LIST_HEAD(&vm->relocated);
>>>     >       INIT_LIST_HEAD(&vm->moved);
>>>     > +     INIT_LIST_HEAD(&vm->dirty);
>>>     >       INIT_LIST_HEAD(&vm->idle);
>>>     >  INIT_LIST_HEAD(&vm->invalidated);
>>>     >  spin_lock_init(&vm->status_lock);
>>>     > @@ -2648,11 +2667,13 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >   {
>>>     >       struct amdgpu_bo_va *bo_va, *tmp;
>>>     >       u64 total_idle = 0;
>>>     > +     u64 total_dirty = 0;
>>>     >       u64 total_relocated = 0;
>>>     >       u64 total_moved = 0;
>>>     >       u64 total_invalidated = 0;
>>>     >       u64 total_done = 0;
>>>     >       unsigned int total_idle_objs = 0;
>>>     > +     unsigned int total_dirty_objs = 0;
>>>     >       unsigned int total_relocated_objs = 0;
>>>     >       unsigned int total_moved_objs = 0;
>>>     >       unsigned int total_invalidated_objs = 0;
>>>     > @@ -2669,6 +2690,15 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >       total_idle_objs = id;
>>>     >       id = 0;
>>>     >
>>>     > +     seq_puts(m, "\tDirty BOs:\n");
>>>     > +     list_for_each_entry_safe(bo_va, tmp, &vm->dirty,
>>>     base.vm_status) {
>>>     > +             if (!bo_va->base.bo <http://base.bo/>)
>>>     > +                     continue;
>>>     > +             total_dirty += amdgpu_bo_print_info(id++,
>>>     bo_va->base.bo <http://base.bo/>, m);
>>>     > +     }
>>>     > +     total_dirty_objs = id;
>>>     > +     id = 0;
>>>     > +
>>>     >       seq_puts(m, "\tRelocated BOs:\n");
>>>     >       list_for_each_entry_safe(bo_va, tmp, &vm->relocated,
>>>     base.vm_status) {
>>>     >               if (!bo_va->base.bo <http://base.bo/>)
>>>     > @@ -2707,6 +2737,8 @@ void amdgpu_debugfs_vm_bo_info(struct
>>>     amdgpu_vm *vm, struct seq_file *m)
>>>     >
>>>     >       seq_printf(m, "\tTotal idle size:  
>>>     %12lld\tobjs:\t%d\n", total_idle,
>>>     >                  total_idle_objs);
>>>     > +     seq_printf(m, "\tTotal dirty size:
>>>      %12lld\tobjs:\t%d\n", total_dirty,
>>>     > +                total_dirty_objs);
>>>     >       seq_printf(m, "\tTotal relocated size:
>>>      %12lld\tobjs:\t%d\n", total_relocated,
>>>     >                  total_relocated_objs);
>>>     >       seq_printf(m, "\tTotal moved size:
>>>      %12lld\tobjs:\t%d\n", total_moved,
>>>     > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > index d9ab97eabda9..f91d4fcf80b8 100644
>>>     > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>     > @@ -276,6 +276,9 @@ struct amdgpu_vm {
>>>     >       /* per VM BOs moved, but not yet updated in the PT */
>>>     >       struct list_head        moved;
>>>     >
>>>     > +     /* normal and per VM BOs that are not moved, but have
>>>     new PT entries */
>>>     > +     struct list_head        dirty;
>>>     > +
>>>     >       /* All BOs of this VM not currently in the state machine */
>>>     >       struct list_head        idle;
>>>     >
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 33080 bytes --]

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm-exynos/exynos-drm-next drm-intel/for-linux-next-fixes linus/master v6.6]
[cannot apply to drm/drm-next drm-intel/for-linux-next drm-tip/drm-tip next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-4-ishitatsuyuki%40gmail.com
patch subject: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311010948.G6I55pTu-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311010948.G6I55pTu-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311010948.G6I55pTu-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:608: warning: Excess function parameter 'bo_va' description in 'amdgpu_gem_va_update_vm'
>> drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:608: warning: Excess function parameter 'operation' description in 'amdgpu_gem_va_update_vm'


vim +608 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c

d38ceaf99ed015f Alex Deucher        2015-04-20  594  
d38ceaf99ed015f Alex Deucher        2015-04-20  595  /**
d38ceaf99ed015f Alex Deucher        2015-04-20  596   * amdgpu_gem_va_update_vm -update the bo_va in its VM
d38ceaf99ed015f Alex Deucher        2015-04-20  597   *
d38ceaf99ed015f Alex Deucher        2015-04-20  598   * @adev: amdgpu_device pointer
dc54d3d1744d23e Christian König     2017-03-13  599   * @vm: vm to update
d38ceaf99ed015f Alex Deucher        2015-04-20  600   * @bo_va: bo_va to update
dc54d3d1744d23e Christian König     2017-03-13  601   * @operation: map, unmap or clear
d38ceaf99ed015f Alex Deucher        2015-04-20  602   *
2ffdaafb5d5f37b Christian König     2017-01-27  603   * Update the bo_va directly after setting its address. Errors are not
d38ceaf99ed015f Alex Deucher        2015-04-20  604   * vital here, so they are not reported back to userspace.
d38ceaf99ed015f Alex Deucher        2015-04-20  605   */
d38ceaf99ed015f Alex Deucher        2015-04-20  606  static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  607  				    struct amdgpu_vm *vm)
d38ceaf99ed015f Alex Deucher        2015-04-20 @608  {
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  609  	struct amdgpu_bo_va *bo_va;
3f3333f8a0e90ac Christian König     2017-08-03  610  	int r;
d38ceaf99ed015f Alex Deucher        2015-04-20  611  
3f3333f8a0e90ac Christian König     2017-08-03  612  	if (!amdgpu_vm_ready(vm))
3f3333f8a0e90ac Christian König     2017-08-03  613  		return;
e410b5cbabe70b1 Chunming Zhou       2015-12-07  614  
f34678187a33970 Nicolai Hähnle      2017-03-23  615  	r = amdgpu_vm_clear_freed(adev, vm, NULL);
d38ceaf99ed015f Alex Deucher        2015-04-20  616  	if (r)
2ffdaafb5d5f37b Christian König     2017-01-27  617  		goto error;
194a33643b1161f monk.liu            2015-07-22  618  
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  619  	spin_lock(&vm->status_lock);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  620  	while (!list_empty(&vm->dirty)) {
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  621  		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  622  					 base.vm_status);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  623  		spin_unlock(&vm->status_lock);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  624  
8f8cc3fb43508a2 Christian König     2022-03-17  625  		r = amdgpu_vm_bo_update(adev, bo_va, false);
0abc6878fc2d699 Christian König     2017-09-01  626  		if (r)
0abc6878fc2d699 Christian König     2017-09-01  627  			goto error;
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  628  		spin_lock(&vm->status_lock);
93bab704c1513f8 Gustavo A. R. Silva 2018-02-14  629  	}
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  630  	spin_unlock(&vm->status_lock);
93bab704c1513f8 Gustavo A. R. Silva 2018-02-14  631  
807e2994092c0bd Christian König     2019-03-14  632  	r = amdgpu_vm_update_pdes(adev, vm, false);
0abc6878fc2d699 Christian König     2017-09-01  633  
2ffdaafb5d5f37b Christian König     2017-01-27  634  error:
68fdd3df79ee4bf Christian König     2015-06-16  635  	if (r && r != -ERESTARTSYS)
d38ceaf99ed015f Alex Deucher        2015-04-20  636  		DRM_ERROR("Couldn't update BO_VA (%d)\n", r);
d38ceaf99ed015f Alex Deucher        2015-04-20  637  }
d38ceaf99ed015f Alex Deucher        2015-04-20  638  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm-exynos/exynos-drm-next drm-intel/for-linux-next-fixes linus/master v6.6]
[cannot apply to drm/drm-next drm-intel/for-linux-next drm-tip/drm-tip next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-4-ishitatsuyuki%40gmail.com
patch subject: [PATCH 3/6] drm/amdgpu: Flush VM updates for split bindings eagerly.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311010948.G6I55pTu-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311010948.G6I55pTu-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311010948.G6I55pTu-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:608: warning: Excess function parameter 'bo_va' description in 'amdgpu_gem_va_update_vm'
>> drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:608: warning: Excess function parameter 'operation' description in 'amdgpu_gem_va_update_vm'


vim +608 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c

d38ceaf99ed015f Alex Deucher        2015-04-20  594  
d38ceaf99ed015f Alex Deucher        2015-04-20  595  /**
d38ceaf99ed015f Alex Deucher        2015-04-20  596   * amdgpu_gem_va_update_vm -update the bo_va in its VM
d38ceaf99ed015f Alex Deucher        2015-04-20  597   *
d38ceaf99ed015f Alex Deucher        2015-04-20  598   * @adev: amdgpu_device pointer
dc54d3d1744d23e Christian König     2017-03-13  599   * @vm: vm to update
d38ceaf99ed015f Alex Deucher        2015-04-20  600   * @bo_va: bo_va to update
dc54d3d1744d23e Christian König     2017-03-13  601   * @operation: map, unmap or clear
d38ceaf99ed015f Alex Deucher        2015-04-20  602   *
2ffdaafb5d5f37b Christian König     2017-01-27  603   * Update the bo_va directly after setting its address. Errors are not
d38ceaf99ed015f Alex Deucher        2015-04-20  604   * vital here, so they are not reported back to userspace.
d38ceaf99ed015f Alex Deucher        2015-04-20  605   */
d38ceaf99ed015f Alex Deucher        2015-04-20  606  static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  607  				    struct amdgpu_vm *vm)
d38ceaf99ed015f Alex Deucher        2015-04-20 @608  {
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  609  	struct amdgpu_bo_va *bo_va;
3f3333f8a0e90ac Christian König     2017-08-03  610  	int r;
d38ceaf99ed015f Alex Deucher        2015-04-20  611  
3f3333f8a0e90ac Christian König     2017-08-03  612  	if (!amdgpu_vm_ready(vm))
3f3333f8a0e90ac Christian König     2017-08-03  613  		return;
e410b5cbabe70b1 Chunming Zhou       2015-12-07  614  
f34678187a33970 Nicolai Hähnle      2017-03-23  615  	r = amdgpu_vm_clear_freed(adev, vm, NULL);
d38ceaf99ed015f Alex Deucher        2015-04-20  616  	if (r)
2ffdaafb5d5f37b Christian König     2017-01-27  617  		goto error;
194a33643b1161f monk.liu            2015-07-22  618  
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  619  	spin_lock(&vm->status_lock);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  620  	while (!list_empty(&vm->dirty)) {
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  621  		bo_va = list_first_entry(&vm->dirty, struct amdgpu_bo_va,
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  622  					 base.vm_status);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  623  		spin_unlock(&vm->status_lock);
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  624  
8f8cc3fb43508a2 Christian König     2022-03-17  625  		r = amdgpu_vm_bo_update(adev, bo_va, false);
0abc6878fc2d699 Christian König     2017-09-01  626  		if (r)
0abc6878fc2d699 Christian König     2017-09-01  627  			goto error;
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  628  		spin_lock(&vm->status_lock);
93bab704c1513f8 Gustavo A. R. Silva 2018-02-14  629  	}
ddf1ffe56ab385a Tatsuyuki Ishi      2023-10-31  630  	spin_unlock(&vm->status_lock);
93bab704c1513f8 Gustavo A. R. Silva 2018-02-14  631  
807e2994092c0bd Christian König     2019-03-14  632  	r = amdgpu_vm_update_pdes(adev, vm, false);
0abc6878fc2d699 Christian König     2017-09-01  633  
2ffdaafb5d5f37b Christian König     2017-01-27  634  error:
68fdd3df79ee4bf Christian König     2015-06-16  635  	if (r && r != -ERESTARTSYS)
d38ceaf99ed015f Alex Deucher        2015-04-20  636  		DRM_ERROR("Couldn't update BO_VA (%d)\n", r);
d38ceaf99ed015f Alex Deucher        2015-04-20  637  }
d38ceaf99ed015f Alex Deucher        2015-04-20  638  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[PATCH 4/6] drm/amdgpu: Remove redundant state change after validation.

[Tatsuyuki Ishi]

All the state changes are handled in the TTM move callback; doing it again
here just leads to more confusion.

The table update remains here because it needs to be done exactly once,
while doing it in the move callback will result it getting triggered twice,
once by the actual BO and once by the shadow BO.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 01d31891cd05..50f7cee639ac 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -495,12 +495,9 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 				return r;
 		}
 
-		if (bo->tbo.type != ttm_bo_type_kernel) {
-			amdgpu_vm_bo_moved(bo_base);
-		} else {
+		if (bo->tbo.type == ttm_bo_type_kernel)
 			vm->update_funcs->map_table(to_amdgpu_bo_vm(bo));
-			amdgpu_vm_bo_relocated(bo_base);
-		}
+
 		spin_lock(&vm->status_lock);
 	}
 	spin_unlock(&vm->status_lock);
-- 
2.42.0

[PATCH 4/6] drm/amdgpu: Remove redundant state change after validation.

[Tatsuyuki Ishi]

All the state changes are handled in the TTM move callback; doing it again
here just leads to more confusion.

The table update remains here because it needs to be done exactly once,
while doing it in the move callback will result it getting triggered twice,
once by the actual BO and once by the shadow BO.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 01d31891cd05..50f7cee639ac 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -495,12 +495,9 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 				return r;
 		}
 
-		if (bo->tbo.type != ttm_bo_type_kernel) {
-			amdgpu_vm_bo_moved(bo_base);
-		} else {
+		if (bo->tbo.type == ttm_bo_type_kernel)
 			vm->update_funcs->map_table(to_amdgpu_bo_vm(bo));
-			amdgpu_vm_bo_relocated(bo_base);
-		}
+
 		spin_lock(&vm->status_lock);
 	}
 	spin_unlock(&vm->status_lock);
-- 
2.42.0

Re: [PATCH 4/6] drm/amdgpu: Remove redundant state change after validation.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> All the state changes are handled in the TTM move callback; doing it again
> here just leads to more confusion.

The state move here is because we need to track which PDs/PTs are 
already validated and which have new locations reflected in the PDEs.

With this change here you will sooner or later run into PDE corruption.

>
> The table update remains here because it needs to be done exactly once,
> while doing it in the move callback will result it getting triggered twice,
> once by the actual BO and once by the shadow BO.

The table update isn't done in the move callback because you can't take 
the appropriate locks there.

Regards,
Christian.


>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 ++-----
>   1 file changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 01d31891cd05..50f7cee639ac 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -495,12 +495,9 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   				return r;
>   		}
>   
> -		if (bo->tbo.type != ttm_bo_type_kernel) {
> -			amdgpu_vm_bo_moved(bo_base);
> -		} else {
> +		if (bo->tbo.type == ttm_bo_type_kernel)
>   			vm->update_funcs->map_table(to_amdgpu_bo_vm(bo));
> -			amdgpu_vm_bo_relocated(bo_base);
> -		}
> +
>   		spin_lock(&vm->status_lock);
>   	}
>   	spin_unlock(&vm->status_lock);

Re: [PATCH 4/6] drm/amdgpu: Remove redundant state change after validation.

[Christian König]

Am 31.10.23 um 14:40 schrieb Tatsuyuki Ishi:
> All the state changes are handled in the TTM move callback; doing it again
> here just leads to more confusion.

The state move here is because we need to track which PDs/PTs are 
already validated and which have new locations reflected in the PDEs.

With this change here you will sooner or later run into PDE corruption.

>
> The table update remains here because it needs to be done exactly once,
> while doing it in the move callback will result it getting triggered twice,
> once by the actual BO and once by the shadow BO.

The table update isn't done in the move callback because you can't take 
the appropriate locks there.

Regards,
Christian.


>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 ++-----
>   1 file changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 01d31891cd05..50f7cee639ac 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -495,12 +495,9 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   				return r;
>   		}
>   
> -		if (bo->tbo.type != ttm_bo_type_kernel) {
> -			amdgpu_vm_bo_moved(bo_base);
> -		} else {
> +		if (bo->tbo.type == ttm_bo_type_kernel)
>   			vm->update_funcs->map_table(to_amdgpu_bo_vm(bo));
> -			amdgpu_vm_bo_relocated(bo_base);
> -		}
> +
>   		spin_lock(&vm->status_lock);
>   	}
>   	spin_unlock(&vm->status_lock);

[PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
Until now, the kernel applied the same rule as implicitly-synchronized
APIs like OpenGL, which with per-VM BOs made page table updates stall the
queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
drivers to opt-out of this behavior, while still ensuring adequate implicit
sync happens for kernel-initiated updates (e.g. BO moves).

We record whether to use implicit sync or not for each freed mapping. To
avoid increasing the mapping struct's size, this is union-ized with the
interval tree field which is unused after the unmap.

The reason this is done with a GEM ioctl flag, instead of being a VM /
context global setting, is that the current libdrm implementation shares
the DRM handle even between different kind of drivers (radeonsi vs radv).

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 +++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 55 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 ++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++---
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 9 files changed, 74 insertions(+), 55 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 7d6daf8d2bfa..10e129bff977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
 	struct amdgpu_device *adev = entry->adev;
 	struct amdgpu_vm *vm = bo_va->base.vm;
 
-	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
+	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
 
 	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 720011019741..612279e65bff 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 		}
 	}
 
-	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
+	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
 	if (r) {
 		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
 		goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 01d3a97248b0..0d9496a06947 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -672,9 +672,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
 		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
 		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
-		AMDGPU_VM_PAGE_NOALLOC;
+		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
 	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
-		AMDGPU_VM_PAGE_PRT;
+		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
 
 	struct drm_amdgpu_gem_va *args = data;
 	struct drm_gem_object *gobj;
@@ -685,6 +685,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	uint64_t va_flags;
 	uint64_t vm_size;
+	bool sync_unmap;
 	int r = 0;
 
 	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
@@ -720,6 +721,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
+
 	switch (args->operation) {
 	case AMDGPU_VA_OP_MAP:
 	case AMDGPU_VA_OP_UNMAP:
@@ -779,19 +782,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 				     va_flags);
 		break;
 	case AMDGPU_VA_OP_UNMAP:
-		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
+		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
+				       sync_unmap);
 		break;
 
 	case AMDGPU_VA_OP_CLEAR:
 		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
 						args->va_address,
-						args->map_size);
+						args->map_size, sync_unmap);
 		break;
 	case AMDGPU_VA_OP_REPLACE:
 		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
 		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
 					     args->offset_in_bo, args->map_size,
-					     va_flags);
+					     va_flags, sync_unmap);
 		break;
 	default:
 		break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index f3ee83cdf97e..28be03f1bbcf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
 	struct rb_node			rb;
 	uint64_t			start;
 	uint64_t			last;
-	uint64_t			__subtree_last;
+	union {
+		/* BOs in interval tree only */
+		uint64_t		__subtree_last;
+		/* Freed BOs only */
+		bool			sync_unmap;
+	};
 	uint64_t			offset;
 	uint64_t			flags;
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 2fd1bfb35916..e71443c8c59b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			     __field(long, last)
 			     __field(u64, offset)
 			     __field(u64, flags)
+			     __field(bool, sync_unmap)
 			     ),
 
 	    TP_fast_assign(
@@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			   __entry->last = mapping->last;
 			   __entry->offset = mapping->offset;
 			   __entry->flags = mapping->flags;
+			   __entry->sync_unmap = mapping->sync_unmap;
 			   ),
-	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
+	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
 		      __entry->bo, __entry->start, __entry->last,
-		      __entry->offset, __entry->flags)
+		      __entry->offset, __entry->flags, __entry->sync_unmap)
 );
 
 DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 50f7cee639ac..a15463e0bbc5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -861,6 +861,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  * @immediate: immediate submission in a page fault
  * @unlocked: unlocked invalidation during MM callback
  * @flush_tlb: trigger tlb invalidation after update completed
+ * @sync_unmap: wait for BO users before unmapping
  * @resv: fences we need to sync to
  * @start: start of mapped range
  * @last: last mapped entry
@@ -878,8 +879,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  */
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence)
 {
@@ -919,7 +921,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
@@ -1165,10 +1167,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 		trace_amdgpu_vm_bo_update(mapping);
 
 		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
-					   resv, mapping->start, mapping->last,
-					   update_flags, mapping->offset,
-					   vram_base, mem, pages_addr,
-					   last_update);
+					   true, resv, mapping->start,
+					   mapping->last, update_flags,
+					   mapping->offset, vram_base, mem,
+					   pages_addr, last_update);
 		if (r)
 			return r;
 	}
@@ -1349,7 +1351,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 		    mapping->start < AMDGPU_GMC_HOLE_START)
 			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
 
-		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
+		r = amdgpu_vm_update_range(adev, vm, false, false, true,
+					   mapping->sync_unmap, resv,
 					   mapping->start, mapping->last,
 					   init_pte_value, 0, 0, NULL, NULL,
 					   &f);
@@ -1589,6 +1592,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * @offset: requested offset in the BO
  * @size: BO size in bytes
  * @flags: attributes of pages (read/write/valid/etc.)
+ * @sync_unmap: wait for BO users before replacing existing mapping
  *
  * Add a mapping of the BO at the specefied addr into the VM. Replace existing
  * mappings as we do so.
@@ -1599,9 +1603,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * Object has to be reserved and unreserved outside!
  */
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t saddr, uint64_t offset,
-			     uint64_t size, uint64_t flags)
+			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_bo *bo = bo_va->base.bo;
@@ -1625,7 +1629,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
 	if (!mapping)
 		return -ENOMEM;
 
-	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
+	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
 	if (r) {
 		kfree(mapping);
 		return r;
@@ -1658,9 +1662,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  *
  * Object has to be reserved and unreserved outside!
  */
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t saddr)
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t saddr, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_vm *vm = bo_va->base.vm;
@@ -1688,6 +1691,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
 	list_del(&mapping->list);
 	amdgpu_vm_it_remove(mapping, &vm->va);
 	mapping->bo_va = NULL;
+	mapping->sync_unmap = sync_unmap;
 	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 
 	if (valid)
@@ -1706,6 +1710,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * @vm: VM structure to use
  * @saddr: start of the range
  * @size: size of the range
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove all mappings in a range, split them as appropriate.
  *
@@ -1713,8 +1718,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * 0 for success, error for failure.
  */
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size)
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
 	LIST_HEAD(removed);
@@ -1782,6 +1787,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 		    tmp->last = eaddr;
 
 		tmp->bo_va = NULL;
+		tmp->sync_unmap = sync_unmap;
 		list_add(&tmp->list, &vm->freed);
 		trace_amdgpu_vm_bo_unmap(NULL, tmp);
 	}
@@ -1899,6 +1905,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 		list_del(&mapping->list);
 		amdgpu_vm_it_remove(mapping, &vm->va);
 		mapping->bo_va = NULL;
+		mapping->sync_unmap = true;
 		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 		list_add(&mapping->list, &vm->freed);
 	}
@@ -2481,20 +2488,19 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 	struct amdgpu_device *adev = drm_to_adev(dev);
 	struct amdgpu_fpriv *fpriv = filp->driver_priv;
 
-	/* No valid flags defined yet */
-	if (args->in.flags)
-		return -EINVAL;
-
 	switch (args->in.op) {
 	case AMDGPU_VM_OP_RESERVE_VMID:
+		if (args->in.flags)
+			return -EINVAL;
 		/* We only have requirement to reserve vmid from gfxhub */
 		if (!fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)]) {
 			amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(0));
 			fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)] = true;
 		}
-
 		break;
 	case AMDGPU_VM_OP_UNRESERVE_VMID:
+		if (args->in.flags)
+			return -EINVAL;
 		if (fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)]) {
 			amdgpu_vmid_free_reserved(adev, AMDGPU_GFXHUB(0));
 			fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)] = false;
@@ -2633,8 +2639,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		goto error_unlock;
 	}
 
-	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
-				   addr, flags, value, 0, NULL, NULL, NULL);
+	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
+				   addr, addr, flags, value, 0, NULL, NULL,
+				   NULL);
 	if (r)
 		goto error_unlock;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index f91d4fcf80b8..3574987595d1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -427,12 +427,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence);
-int amdgpu_vm_bo_update(struct amdgpu_device *adev,
-			struct amdgpu_bo_va *bo_va,
+int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			bool clear);
 bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
 void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
@@ -448,15 +448,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
 		     uint64_t addr, uint64_t offset,
 		     uint64_t size, uint64_t flags);
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t addr, uint64_t offset,
-			     uint64_t size, uint64_t flags);
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t addr);
+			     struct amdgpu_bo_va *bo_va, uint64_t addr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap);
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t addr, bool sync_unmap);
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size);
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap);
 struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
 							 uint64_t addr);
 void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index bb16b795d1bc..6eb4a0a4bc84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	pr_debug("[0x%llx 0x%llx]\n", start, last);
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
-				      fence);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
+				      start, last, init_pte_value, 0, 0, NULL,
+				      NULL, fence);
 }
 
 static int
@@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		 * different memory partition based on fpfn/lpfn, we should use
 		 * same vm_manager.vram_base_offset regardless memory partition.
 		 */
-		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
-					   last_start, prange->start + i,
-					   pte_flags,
-					   (last_start - prange->start) << PAGE_SHIFT,
-					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
-					   NULL, dma_addr, &vm->last_update);
+		r = amdgpu_vm_update_range(
+			adev, vm, false, false, flush_tlb, true, NULL,
+			last_start, prange->start + i, pte_flags,
+			(last_start - prange->start) << PAGE_SHIFT,
+			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
+			NULL, dma_addr, &vm->last_update);
 
 		for (j = last_start - prange->start; j <= i; j++)
 			dma_addr[j] |= last_domain;
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index f477eda6a2b8..3cdcc299956e 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
 #define AMDGPU_VM_MTYPE_RW		(5 << 5)
 /* don't allocate MALL */
 #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
+/* don't sync on unmap */
+#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
 
 struct drm_amdgpu_gem_va {
 	/** GEM object handle */
-- 
2.42.0

[PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
Until now, the kernel applied the same rule as implicitly-synchronized
APIs like OpenGL, which with per-VM BOs made page table updates stall the
queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
drivers to opt-out of this behavior, while still ensuring adequate implicit
sync happens for kernel-initiated updates (e.g. BO moves).

We record whether to use implicit sync or not for each freed mapping. To
avoid increasing the mapping struct's size, this is union-ized with the
interval tree field which is unused after the unmap.

The reason this is done with a GEM ioctl flag, instead of being a VM /
context global setting, is that the current libdrm implementation shares
the DRM handle even between different kind of drivers (radeonsi vs radv).

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 +++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 55 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 ++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++---
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 9 files changed, 74 insertions(+), 55 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 7d6daf8d2bfa..10e129bff977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
 	struct amdgpu_device *adev = entry->adev;
 	struct amdgpu_vm *vm = bo_va->base.vm;
 
-	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
+	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
 
 	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 720011019741..612279e65bff 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 		}
 	}
 
-	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
+	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
 	if (r) {
 		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
 		goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 01d3a97248b0..0d9496a06947 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -672,9 +672,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
 		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
 		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
-		AMDGPU_VM_PAGE_NOALLOC;
+		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
 	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
-		AMDGPU_VM_PAGE_PRT;
+		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
 
 	struct drm_amdgpu_gem_va *args = data;
 	struct drm_gem_object *gobj;
@@ -685,6 +685,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	uint64_t va_flags;
 	uint64_t vm_size;
+	bool sync_unmap;
 	int r = 0;
 
 	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
@@ -720,6 +721,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
+
 	switch (args->operation) {
 	case AMDGPU_VA_OP_MAP:
 	case AMDGPU_VA_OP_UNMAP:
@@ -779,19 +782,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 				     va_flags);
 		break;
 	case AMDGPU_VA_OP_UNMAP:
-		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
+		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
+				       sync_unmap);
 		break;
 
 	case AMDGPU_VA_OP_CLEAR:
 		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
 						args->va_address,
-						args->map_size);
+						args->map_size, sync_unmap);
 		break;
 	case AMDGPU_VA_OP_REPLACE:
 		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
 		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
 					     args->offset_in_bo, args->map_size,
-					     va_flags);
+					     va_flags, sync_unmap);
 		break;
 	default:
 		break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index f3ee83cdf97e..28be03f1bbcf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
 	struct rb_node			rb;
 	uint64_t			start;
 	uint64_t			last;
-	uint64_t			__subtree_last;
+	union {
+		/* BOs in interval tree only */
+		uint64_t		__subtree_last;
+		/* Freed BOs only */
+		bool			sync_unmap;
+	};
 	uint64_t			offset;
 	uint64_t			flags;
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 2fd1bfb35916..e71443c8c59b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			     __field(long, last)
 			     __field(u64, offset)
 			     __field(u64, flags)
+			     __field(bool, sync_unmap)
 			     ),
 
 	    TP_fast_assign(
@@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			   __entry->last = mapping->last;
 			   __entry->offset = mapping->offset;
 			   __entry->flags = mapping->flags;
+			   __entry->sync_unmap = mapping->sync_unmap;
 			   ),
-	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
+	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
 		      __entry->bo, __entry->start, __entry->last,
-		      __entry->offset, __entry->flags)
+		      __entry->offset, __entry->flags, __entry->sync_unmap)
 );
 
 DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 50f7cee639ac..a15463e0bbc5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -861,6 +861,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  * @immediate: immediate submission in a page fault
  * @unlocked: unlocked invalidation during MM callback
  * @flush_tlb: trigger tlb invalidation after update completed
+ * @sync_unmap: wait for BO users before unmapping
  * @resv: fences we need to sync to
  * @start: start of mapped range
  * @last: last mapped entry
@@ -878,8 +879,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  */
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence)
 {
@@ -919,7 +921,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
@@ -1165,10 +1167,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 		trace_amdgpu_vm_bo_update(mapping);
 
 		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
-					   resv, mapping->start, mapping->last,
-					   update_flags, mapping->offset,
-					   vram_base, mem, pages_addr,
-					   last_update);
+					   true, resv, mapping->start,
+					   mapping->last, update_flags,
+					   mapping->offset, vram_base, mem,
+					   pages_addr, last_update);
 		if (r)
 			return r;
 	}
@@ -1349,7 +1351,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 		    mapping->start < AMDGPU_GMC_HOLE_START)
 			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
 
-		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
+		r = amdgpu_vm_update_range(adev, vm, false, false, true,
+					   mapping->sync_unmap, resv,
 					   mapping->start, mapping->last,
 					   init_pte_value, 0, 0, NULL, NULL,
 					   &f);
@@ -1589,6 +1592,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * @offset: requested offset in the BO
  * @size: BO size in bytes
  * @flags: attributes of pages (read/write/valid/etc.)
+ * @sync_unmap: wait for BO users before replacing existing mapping
  *
  * Add a mapping of the BO at the specefied addr into the VM. Replace existing
  * mappings as we do so.
@@ -1599,9 +1603,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * Object has to be reserved and unreserved outside!
  */
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t saddr, uint64_t offset,
-			     uint64_t size, uint64_t flags)
+			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_bo *bo = bo_va->base.bo;
@@ -1625,7 +1629,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
 	if (!mapping)
 		return -ENOMEM;
 
-	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
+	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
 	if (r) {
 		kfree(mapping);
 		return r;
@@ -1658,9 +1662,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  *
  * Object has to be reserved and unreserved outside!
  */
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t saddr)
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t saddr, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_vm *vm = bo_va->base.vm;
@@ -1688,6 +1691,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
 	list_del(&mapping->list);
 	amdgpu_vm_it_remove(mapping, &vm->va);
 	mapping->bo_va = NULL;
+	mapping->sync_unmap = sync_unmap;
 	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 
 	if (valid)
@@ -1706,6 +1710,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * @vm: VM structure to use
  * @saddr: start of the range
  * @size: size of the range
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove all mappings in a range, split them as appropriate.
  *
@@ -1713,8 +1718,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * 0 for success, error for failure.
  */
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size)
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
 	LIST_HEAD(removed);
@@ -1782,6 +1787,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 		    tmp->last = eaddr;
 
 		tmp->bo_va = NULL;
+		tmp->sync_unmap = sync_unmap;
 		list_add(&tmp->list, &vm->freed);
 		trace_amdgpu_vm_bo_unmap(NULL, tmp);
 	}
@@ -1899,6 +1905,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 		list_del(&mapping->list);
 		amdgpu_vm_it_remove(mapping, &vm->va);
 		mapping->bo_va = NULL;
+		mapping->sync_unmap = true;
 		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 		list_add(&mapping->list, &vm->freed);
 	}
@@ -2481,20 +2488,19 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 	struct amdgpu_device *adev = drm_to_adev(dev);
 	struct amdgpu_fpriv *fpriv = filp->driver_priv;
 
-	/* No valid flags defined yet */
-	if (args->in.flags)
-		return -EINVAL;
-
 	switch (args->in.op) {
 	case AMDGPU_VM_OP_RESERVE_VMID:
+		if (args->in.flags)
+			return -EINVAL;
 		/* We only have requirement to reserve vmid from gfxhub */
 		if (!fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)]) {
 			amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(0));
 			fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)] = true;
 		}
-
 		break;
 	case AMDGPU_VM_OP_UNRESERVE_VMID:
+		if (args->in.flags)
+			return -EINVAL;
 		if (fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)]) {
 			amdgpu_vmid_free_reserved(adev, AMDGPU_GFXHUB(0));
 			fpriv->vm.reserved_vmid[AMDGPU_GFXHUB(0)] = false;
@@ -2633,8 +2639,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		goto error_unlock;
 	}
 
-	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
-				   addr, flags, value, 0, NULL, NULL, NULL);
+	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
+				   addr, addr, flags, value, 0, NULL, NULL,
+				   NULL);
 	if (r)
 		goto error_unlock;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index f91d4fcf80b8..3574987595d1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -427,12 +427,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence);
-int amdgpu_vm_bo_update(struct amdgpu_device *adev,
-			struct amdgpu_bo_va *bo_va,
+int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			bool clear);
 bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
 void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
@@ -448,15 +448,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
 		     uint64_t addr, uint64_t offset,
 		     uint64_t size, uint64_t flags);
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t addr, uint64_t offset,
-			     uint64_t size, uint64_t flags);
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t addr);
+			     struct amdgpu_bo_va *bo_va, uint64_t addr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap);
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t addr, bool sync_unmap);
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size);
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap);
 struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
 							 uint64_t addr);
 void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index bb16b795d1bc..6eb4a0a4bc84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	pr_debug("[0x%llx 0x%llx]\n", start, last);
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
-				      fence);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
+				      start, last, init_pte_value, 0, 0, NULL,
+				      NULL, fence);
 }
 
 static int
@@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		 * different memory partition based on fpfn/lpfn, we should use
 		 * same vm_manager.vram_base_offset regardless memory partition.
 		 */
-		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
-					   last_start, prange->start + i,
-					   pte_flags,
-					   (last_start - prange->start) << PAGE_SHIFT,
-					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
-					   NULL, dma_addr, &vm->last_update);
+		r = amdgpu_vm_update_range(
+			adev, vm, false, false, flush_tlb, true, NULL,
+			last_start, prange->start + i, pte_flags,
+			(last_start - prange->start) << PAGE_SHIFT,
+			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
+			NULL, dma_addr, &vm->last_update);
 
 		for (j = last_start - prange->start; j <= i; j++)
 			dma_addr[j] |= last_domain;
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index f477eda6a2b8..3cdcc299956e 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
 #define AMDGPU_VM_MTYPE_RW		(5 << 5)
 /* don't allocate MALL */
 #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
+/* don't sync on unmap */
+#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
 
 struct drm_amdgpu_gem_va {
 	/** GEM object handle */
-- 
2.42.0

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Michel Dänzer]

On 10/31/23 14:40, Tatsuyuki Ishi wrote:
> In Vulkan, it is the application's responsibility to perform adequate
> synchronization before a sparse unmap, replace or BO destroy operation.
> Until now, the kernel applied the same rule as implicitly-synchronized
> APIs like OpenGL, which with per-VM BOs made page table updates stall the
> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> drivers to opt-out of this behavior, while still ensuring adequate implicit
> sync happens for kernel-initiated updates (e.g. BO moves).
> 
> We record whether to use implicit sync or not for each freed mapping. To
> avoid increasing the mapping struct's size, this is union-ized with the
> interval tree field which is unused after the unmap.
> 
> The reason this is done with a GEM ioctl flag, instead of being a VM /
> context global setting, is that the current libdrm implementation shares
> the DRM handle even between different kind of drivers (radeonsi vs radv).

Different drivers always use separate contexts though, even with the same DRM file description, don't they?

FWIW, RADV will also want explicit sync in the CS ioctl.


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Michel Dänzer]

On 10/31/23 14:40, Tatsuyuki Ishi wrote:
> In Vulkan, it is the application's responsibility to perform adequate
> synchronization before a sparse unmap, replace or BO destroy operation.
> Until now, the kernel applied the same rule as implicitly-synchronized
> APIs like OpenGL, which with per-VM BOs made page table updates stall the
> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> drivers to opt-out of this behavior, while still ensuring adequate implicit
> sync happens for kernel-initiated updates (e.g. BO moves).
> 
> We record whether to use implicit sync or not for each freed mapping. To
> avoid increasing the mapping struct's size, this is union-ized with the
> interval tree field which is unused after the unmap.
> 
> The reason this is done with a GEM ioctl flag, instead of being a VM /
> context global setting, is that the current libdrm implementation shares
> the DRM handle even between different kind of drivers (radeonsi vs radv).

Different drivers always use separate contexts though, even with the same DRM file description, don't they?

FWIW, RADV will also want explicit sync in the CS ioctl.


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Tue, Oct 31, 2023 at 3:14 PM Michel Dänzer <michel.daenzer@mailbox.org>
wrote:

> On 10/31/23 14:40, Tatsuyuki Ishi wrote:
> > In Vulkan, it is the application's responsibility to perform adequate
> > synchronization before a sparse unmap, replace or BO destroy operation.
> > Until now, the kernel applied the same rule as implicitly-synchronized
> > APIs like OpenGL, which with per-VM BOs made page table updates stall the
> > queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> > drivers to opt-out of this behavior, while still ensuring adequate
> implicit
> > sync happens for kernel-initiated updates (e.g. BO moves).
> >
> > We record whether to use implicit sync or not for each freed mapping. To
> > avoid increasing the mapping struct's size, this is union-ized with the
> > interval tree field which is unused after the unmap.
> >
> > The reason this is done with a GEM ioctl flag, instead of being a VM /
> > context global setting, is that the current libdrm implementation shares
> > the DRM handle even between different kind of drivers (radeonsi vs radv).
>
> Different drivers always use separate contexts though, even with the same
> DRM file description, don't they?
>
> FWIW, RADV will also want explicit sync in the CS ioctl.
>
> I think a crucial problem is that VA ioctls don't take a context so a
per-context flag doesn't solve this (the previous attempt used it because
all the sync changes were on the CS submit side and not the VA ioctl side)
. So I'd still like to solve that side for RADV, but I think the VA ioctl
flag makes sense here if we need to do anything different VA ioctl wise.


> --
> Earthling Michel Dänzer            |                  https://redhat.com
> Libre software enthusiast          |         Mesa and Xwayland developer
>
>

[-- Attachment #2: Type: text/html, Size: 2516 bytes --]

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Bas Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Tue, Oct 31, 2023 at 3:14 PM Michel Dänzer <michel.daenzer@mailbox.org>
wrote:

> On 10/31/23 14:40, Tatsuyuki Ishi wrote:
> > In Vulkan, it is the application's responsibility to perform adequate
> > synchronization before a sparse unmap, replace or BO destroy operation.
> > Until now, the kernel applied the same rule as implicitly-synchronized
> > APIs like OpenGL, which with per-VM BOs made page table updates stall the
> > queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> > drivers to opt-out of this behavior, while still ensuring adequate
> implicit
> > sync happens for kernel-initiated updates (e.g. BO moves).
> >
> > We record whether to use implicit sync or not for each freed mapping. To
> > avoid increasing the mapping struct's size, this is union-ized with the
> > interval tree field which is unused after the unmap.
> >
> > The reason this is done with a GEM ioctl flag, instead of being a VM /
> > context global setting, is that the current libdrm implementation shares
> > the DRM handle even between different kind of drivers (radeonsi vs radv).
>
> Different drivers always use separate contexts though, even with the same
> DRM file description, don't they?
>
> FWIW, RADV will also want explicit sync in the CS ioctl.
>
> I think a crucial problem is that VA ioctls don't take a context so a
per-context flag doesn't solve this (the previous attempt used it because
all the sync changes were on the CS submit side and not the VA ioctl side)
. So I'd still like to solve that side for RADV, but I think the VA ioctl
flag makes sense here if we need to do anything different VA ioctl wise.


> --
> Earthling Michel Dänzer            |                  https://redhat.com
> Libre software enthusiast          |         Mesa and Xwayland developer
>
>

[-- Attachment #2: Type: text/html, Size: 2516 bytes --]

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]



Am 31.10.23 um 15:14 schrieb Michel Dänzer:
> On 10/31/23 14:40, Tatsuyuki Ishi wrote:
>> In Vulkan, it is the application's responsibility to perform adequate
>> synchronization before a sparse unmap, replace or BO destroy operation.
>> Until now, the kernel applied the same rule as implicitly-synchronized
>> APIs like OpenGL, which with per-VM BOs made page table updates stall the
>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>> drivers to opt-out of this behavior, while still ensuring adequate implicit
>> sync happens for kernel-initiated updates (e.g. BO moves).
>>
>> We record whether to use implicit sync or not for each freed mapping. To
>> avoid increasing the mapping struct's size, this is union-ized with the
>> interval tree field which is unused after the unmap.
>>
>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>> context global setting, is that the current libdrm implementation shares
>> the DRM handle even between different kind of drivers (radeonsi vs radv).
> Different drivers always use separate contexts though, even with the same DRM file description, don't they?

Separate contexts don't help here since the VA space is shared between 
the two.

>
> FWIW, RADV will also want explicit sync in the CS ioctl.
You can replace that with the DMA-buf IOCTLs like Faith is planning to 
do for NVK. Regards, Christian.

[-- Attachment #2: Type: text/html, Size: 2093 bytes --]

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]



Am 31.10.23 um 15:14 schrieb Michel Dänzer:
> On 10/31/23 14:40, Tatsuyuki Ishi wrote:
>> In Vulkan, it is the application's responsibility to perform adequate
>> synchronization before a sparse unmap, replace or BO destroy operation.
>> Until now, the kernel applied the same rule as implicitly-synchronized
>> APIs like OpenGL, which with per-VM BOs made page table updates stall the
>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>> drivers to opt-out of this behavior, while still ensuring adequate implicit
>> sync happens for kernel-initiated updates (e.g. BO moves).
>>
>> We record whether to use implicit sync or not for each freed mapping. To
>> avoid increasing the mapping struct's size, this is union-ized with the
>> interval tree field which is unused after the unmap.
>>
>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>> context global setting, is that the current libdrm implementation shares
>> the DRM handle even between different kind of drivers (radeonsi vs radv).
> Different drivers always use separate contexts though, even with the same DRM file description, don't they?

Separate contexts don't help here since the VA space is shared between 
the two.

>
> FWIW, RADV will also want explicit sync in the CS ioctl.
You can replace that with the DMA-buf IOCTLs like Faith is planning to 
do for NVK. Regards, Christian.

[-- Attachment #2: Type: text/html, Size: 2093 bytes --]

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Michel Dänzer]

On 10/31/23 15:34, Christian König wrote:
> Am 31.10.23 um 15:14 schrieb Michel Dänzer:
> 
>> FWIW, RADV will also want explicit sync in the CS ioctl.
> You can replace that with the DMA-buf IOCTLs like Faith is planning to do for NVK. 

Those ioctls cannot disable implicit sync for the CS ioctl. They can be used for making implicit sync work correctly for individual BOs though, once implicit sync is disabled for the CS ioctl.


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Michel Dänzer]

On 10/31/23 15:34, Christian König wrote:
> Am 31.10.23 um 15:14 schrieb Michel Dänzer:
> 
>> FWIW, RADV will also want explicit sync in the CS ioctl.
> You can replace that with the DMA-buf IOCTLs like Faith is planning to do for NVK. 

Those ioctls cannot disable implicit sync for the CS ioctl. They can be used for making implicit sync work correctly for individual BOs though, once implicit sync is disabled for the CS ioctl.


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm-exynos/exynos-drm-next drm-intel/for-linux-next-fixes linus/master v6.6]
[cannot apply to drm/drm-next drm-intel/for-linux-next drm-tip/drm-tip next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-6-ishitatsuyuki%40gmail.com
patch subject: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311011037.Bt6NSYwA-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311011037.Bt6NSYwA-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311011037.Bt6NSYwA-lkp@intel.com/

All warnings (new ones prefixed by >>):

   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: Function parameter or member 'evicted' not described in 'amdgpu_vm_bo_set_evicted'
   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: expecting prototype for amdgpu_vm_bo_evicted(). Prototype was for amdgpu_vm_bo_set_evicted() instead
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:1667: warning: Function parameter or member 'sync_unmap' not described in 'amdgpu_vm_bo_unmap'


vim +1667 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

d38ceaf99ed015 Alex Deucher      2015-04-20  1650  
d38ceaf99ed015 Alex Deucher      2015-04-20  1651  /**
d38ceaf99ed015 Alex Deucher      2015-04-20  1652   * amdgpu_vm_bo_unmap - remove bo mapping from vm
d38ceaf99ed015 Alex Deucher      2015-04-20  1653   *
d38ceaf99ed015 Alex Deucher      2015-04-20  1654   * @adev: amdgpu_device pointer
d38ceaf99ed015 Alex Deucher      2015-04-20  1655   * @bo_va: bo_va to remove the address from
d38ceaf99ed015 Alex Deucher      2015-04-20  1656   * @saddr: where to the BO is mapped
d38ceaf99ed015 Alex Deucher      2015-04-20  1657   *
d38ceaf99ed015 Alex Deucher      2015-04-20  1658   * Remove a mapping of the BO at the specefied addr from the VM.
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1659   *
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1660   * Returns:
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1661   * 0 for success, error for failure.
d38ceaf99ed015 Alex Deucher      2015-04-20  1662   *
49b02b180a541d Chunming Zhou     2015-11-13  1663   * Object has to be reserved and unreserved outside!
d38ceaf99ed015 Alex Deucher      2015-04-20  1664   */
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1665  int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1666  		       uint64_t saddr, bool sync_unmap)
d38ceaf99ed015 Alex Deucher      2015-04-20 @1667  {
d38ceaf99ed015 Alex Deucher      2015-04-20  1668  	struct amdgpu_bo_va_mapping *mapping;
ec681545afe5a4 Christian König   2017-08-01  1669  	struct amdgpu_vm *vm = bo_va->base.vm;
7fc11959018f8b Christian König   2015-07-30  1670  	bool valid = true;
d38ceaf99ed015 Alex Deucher      2015-04-20  1671  
6c7fc503a47f9b Christian König   2015-06-05  1672  	saddr /= AMDGPU_GPU_PAGE_SIZE;
32b41ac21fde8f Christian König   2016-03-08  1673  
7fc11959018f8b Christian König   2015-07-30  1674  	list_for_each_entry(mapping, &bo_va->valids, list) {
a9f87f64525435 Christian König   2017-03-30  1675  		if (mapping->start == saddr)
7fc11959018f8b Christian König   2015-07-30  1676  			break;
7fc11959018f8b Christian König   2015-07-30  1677  	}
7fc11959018f8b Christian König   2015-07-30  1678  
7fc11959018f8b Christian König   2015-07-30  1679  	if (&mapping->list == &bo_va->valids) {
7fc11959018f8b Christian König   2015-07-30  1680  		valid = false;
7fc11959018f8b Christian König   2015-07-30  1681  
7fc11959018f8b Christian König   2015-07-30  1682  		list_for_each_entry(mapping, &bo_va->invalids, list) {
a9f87f64525435 Christian König   2017-03-30  1683  			if (mapping->start == saddr)
d38ceaf99ed015 Alex Deucher      2015-04-20  1684  				break;
d38ceaf99ed015 Alex Deucher      2015-04-20  1685  		}
d38ceaf99ed015 Alex Deucher      2015-04-20  1686  
32b41ac21fde8f Christian König   2016-03-08  1687  		if (&mapping->list == &bo_va->invalids)
d38ceaf99ed015 Alex Deucher      2015-04-20  1688  			return -ENOENT;
d38ceaf99ed015 Alex Deucher      2015-04-20  1689  	}
32b41ac21fde8f Christian König   2016-03-08  1690  
d38ceaf99ed015 Alex Deucher      2015-04-20  1691  	list_del(&mapping->list);
a9f87f64525435 Christian König   2017-03-30  1692  	amdgpu_vm_it_remove(mapping, &vm->va);
aebc5e6f50f770 Christian König   2017-09-06  1693  	mapping->bo_va = NULL;
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1694  	mapping->sync_unmap = sync_unmap;
93e3e4385b69d8 Christian König   2015-06-09  1695  	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
d38ceaf99ed015 Alex Deucher      2015-04-20  1696  
e17841b97587ad Christian König   2016-03-08  1697  	if (valid)
d38ceaf99ed015 Alex Deucher      2015-04-20  1698  		list_add(&mapping->list, &vm->freed);
e17841b97587ad Christian König   2016-03-08  1699  	else
284710fa6c3a5f Christian König   2017-01-30  1700  		amdgpu_vm_free_mapping(adev, vm, mapping,
284710fa6c3a5f Christian König   2017-01-30  1701  				       bo_va->last_pt_update);
d38ceaf99ed015 Alex Deucher      2015-04-20  1702  
d38ceaf99ed015 Alex Deucher      2015-04-20  1703  	return 0;
d38ceaf99ed015 Alex Deucher      2015-04-20  1704  }
d38ceaf99ed015 Alex Deucher      2015-04-20  1705  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[kernel test robot]

Hi Tatsuyuki,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-misc/drm-misc-next]
[also build test WARNING on drm-exynos/exynos-drm-next drm-intel/for-linux-next-fixes linus/master v6.6]
[cannot apply to drm/drm-next drm-intel/for-linux-next drm-tip/drm-tip next-20231031]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Tatsuyuki-Ishi/drm-amdgpu-Don-t-implicit-sync-PRT-maps/20231031-224530
base:   git://anongit.freedesktop.org/drm/drm-misc drm-misc-next
patch link:    https://lore.kernel.org/r/20231031134059.171277-6-ishitatsuyuki%40gmail.com
patch subject: [PATCH 5/6] drm/amdgpu: Add flag to disable implicit sync for GEM operations.
config: arc-randconfig-001-20231101 (https://download.01.org/0day-ci/archive/20231101/202311011037.Bt6NSYwA-lkp@intel.com/config)
compiler: arceb-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231101/202311011037.Bt6NSYwA-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202311011037.Bt6NSYwA-lkp@intel.com/

All warnings (new ones prefixed by >>):

   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: Function parameter or member 'evicted' not described in 'amdgpu_vm_bo_set_evicted'
   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:178: warning: expecting prototype for amdgpu_vm_bo_evicted(). Prototype was for amdgpu_vm_bo_set_evicted() instead
>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c:1667: warning: Function parameter or member 'sync_unmap' not described in 'amdgpu_vm_bo_unmap'


vim +1667 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c

d38ceaf99ed015 Alex Deucher      2015-04-20  1650  
d38ceaf99ed015 Alex Deucher      2015-04-20  1651  /**
d38ceaf99ed015 Alex Deucher      2015-04-20  1652   * amdgpu_vm_bo_unmap - remove bo mapping from vm
d38ceaf99ed015 Alex Deucher      2015-04-20  1653   *
d38ceaf99ed015 Alex Deucher      2015-04-20  1654   * @adev: amdgpu_device pointer
d38ceaf99ed015 Alex Deucher      2015-04-20  1655   * @bo_va: bo_va to remove the address from
d38ceaf99ed015 Alex Deucher      2015-04-20  1656   * @saddr: where to the BO is mapped
d38ceaf99ed015 Alex Deucher      2015-04-20  1657   *
d38ceaf99ed015 Alex Deucher      2015-04-20  1658   * Remove a mapping of the BO at the specefied addr from the VM.
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1659   *
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1660   * Returns:
7fc48e5912795c Andrey Grodzovsky 2018-06-11  1661   * 0 for success, error for failure.
d38ceaf99ed015 Alex Deucher      2015-04-20  1662   *
49b02b180a541d Chunming Zhou     2015-11-13  1663   * Object has to be reserved and unreserved outside!
d38ceaf99ed015 Alex Deucher      2015-04-20  1664   */
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1665  int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1666  		       uint64_t saddr, bool sync_unmap)
d38ceaf99ed015 Alex Deucher      2015-04-20 @1667  {
d38ceaf99ed015 Alex Deucher      2015-04-20  1668  	struct amdgpu_bo_va_mapping *mapping;
ec681545afe5a4 Christian König   2017-08-01  1669  	struct amdgpu_vm *vm = bo_va->base.vm;
7fc11959018f8b Christian König   2015-07-30  1670  	bool valid = true;
d38ceaf99ed015 Alex Deucher      2015-04-20  1671  
6c7fc503a47f9b Christian König   2015-06-05  1672  	saddr /= AMDGPU_GPU_PAGE_SIZE;
32b41ac21fde8f Christian König   2016-03-08  1673  
7fc11959018f8b Christian König   2015-07-30  1674  	list_for_each_entry(mapping, &bo_va->valids, list) {
a9f87f64525435 Christian König   2017-03-30  1675  		if (mapping->start == saddr)
7fc11959018f8b Christian König   2015-07-30  1676  			break;
7fc11959018f8b Christian König   2015-07-30  1677  	}
7fc11959018f8b Christian König   2015-07-30  1678  
7fc11959018f8b Christian König   2015-07-30  1679  	if (&mapping->list == &bo_va->valids) {
7fc11959018f8b Christian König   2015-07-30  1680  		valid = false;
7fc11959018f8b Christian König   2015-07-30  1681  
7fc11959018f8b Christian König   2015-07-30  1682  		list_for_each_entry(mapping, &bo_va->invalids, list) {
a9f87f64525435 Christian König   2017-03-30  1683  			if (mapping->start == saddr)
d38ceaf99ed015 Alex Deucher      2015-04-20  1684  				break;
d38ceaf99ed015 Alex Deucher      2015-04-20  1685  		}
d38ceaf99ed015 Alex Deucher      2015-04-20  1686  
32b41ac21fde8f Christian König   2016-03-08  1687  		if (&mapping->list == &bo_va->invalids)
d38ceaf99ed015 Alex Deucher      2015-04-20  1688  			return -ENOENT;
d38ceaf99ed015 Alex Deucher      2015-04-20  1689  	}
32b41ac21fde8f Christian König   2016-03-08  1690  
d38ceaf99ed015 Alex Deucher      2015-04-20  1691  	list_del(&mapping->list);
a9f87f64525435 Christian König   2017-03-30  1692  	amdgpu_vm_it_remove(mapping, &vm->va);
aebc5e6f50f770 Christian König   2017-09-06  1693  	mapping->bo_va = NULL;
1550024e9de031 Tatsuyuki Ishi    2023-10-31  1694  	mapping->sync_unmap = sync_unmap;
93e3e4385b69d8 Christian König   2015-06-09  1695  	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
d38ceaf99ed015 Alex Deucher      2015-04-20  1696  
e17841b97587ad Christian König   2016-03-08  1697  	if (valid)
d38ceaf99ed015 Alex Deucher      2015-04-20  1698  		list_add(&mapping->list, &vm->freed);
e17841b97587ad Christian König   2016-03-08  1699  	else
284710fa6c3a5f Christian König   2017-01-30  1700  		amdgpu_vm_free_mapping(adev, vm, mapping,
284710fa6c3a5f Christian König   2017-01-30  1701  				       bo_va->last_pt_update);
d38ceaf99ed015 Alex Deucher      2015-04-20  1702  
d38ceaf99ed015 Alex Deucher      2015-04-20  1703  	return 0;
d38ceaf99ed015 Alex Deucher      2015-04-20  1704  }
d38ceaf99ed015 Alex Deucher      2015-04-20  1705  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[PATCH 6/6] drm/amdgpu: Bump amdgpu driver version.

[Tatsuyuki Ishi]

For detection of the new explicit sync functionality without having to try
the ioctl.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 81edf66dbea8..2aa406dee192 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -113,9 +113,10 @@
  *            gl1c_cache_size, gl2c_cache_size, mall_size, enabled_rb_pipes_mask_hi
  *   3.53.0 - Support for GFX11 CP GFX shadowing
  *   3.54.0 - Add AMDGPU_CTX_QUERY2_FLAGS_RESET_IN_PROGRESS support
+ * - 3.55.0 - Add AMDGPU_VM_EXPLICIT_SYNC flag for GEM operations.
  */
 #define KMS_DRIVER_MAJOR	3
-#define KMS_DRIVER_MINOR	54
+#define KMS_DRIVER_MINOR	55
 #define KMS_DRIVER_PATCHLEVEL	0
 
 unsigned int amdgpu_vram_limit = UINT_MAX;
-- 
2.42.0

[PATCH 6/6] drm/amdgpu: Bump amdgpu driver version.

[Tatsuyuki Ishi]

For detection of the new explicit sync functionality without having to try
the ioctl.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 81edf66dbea8..2aa406dee192 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -113,9 +113,10 @@
  *            gl1c_cache_size, gl2c_cache_size, mall_size, enabled_rb_pipes_mask_hi
  *   3.53.0 - Support for GFX11 CP GFX shadowing
  *   3.54.0 - Add AMDGPU_CTX_QUERY2_FLAGS_RESET_IN_PROGRESS support
+ * - 3.55.0 - Add AMDGPU_VM_EXPLICIT_SYNC flag for GEM operations.
  */
 #define KMS_DRIVER_MAJOR	3
-#define KMS_DRIVER_MINOR	54
+#define KMS_DRIVER_MINOR	55
 #define KMS_DRIVER_PATCHLEVEL	0
 
 unsigned int amdgpu_vram_limit = UINT_MAX;
-- 
2.42.0

[PATCH v2 0/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
This adds an option to AMDGPU_VA_OPs to disable redundant implicit sync
that happens on sparse unmap or replace operations.

This has seen a significant improvement in stutter in Forza Horizon 5
and Forza Horizon 4. (As games that had significant issues in sparse
binding related stutter).

Compared to the previous series [1], this specifically targets the VM
operations and keep everything else intact, including implicit sync on
kernel-initiated moves.

I've been able to pass a full Vulkan CTS run on Navi 10 with this.

Userspace code for this is available at [2] and a branch for the kernel
code is available at [3].

v2 changes:
- Drop the changes to flush split bindings eagerly as its incompatible
  with TLB flush quirks in current hardware. Drop the refactoring
  commits related to that change too.
- Fixed a missing doc warning.
- Removed an accidentally included ioctl change.

[1]: https://lore.kernel.org/all/20230821062005.109771-1-ishitatsuyuki@gmail.com/
[2]: https://gitlab.freedesktop.org/ishitatsuyuki/mesa/-/commits/vm-explicit-sync
[3]: https://github.com/ishitatsuyuki/linux/tree/explicit-sync-drm-misc-next

Tatsuyuki Ishi (3):
  drm/amdgpu: Don't implicit sync PRT maps.
  drm/amdgpu: Add flag to disable implicit sync for GEM operations.
  drm/amdgpu: Bump amdgpu driver version.

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 10 files changed, 73 insertions(+), 51 deletions(-)

-- 
2.42.0

[PATCH v2 0/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
This adds an option to AMDGPU_VA_OPs to disable redundant implicit sync
that happens on sparse unmap or replace operations.

This has seen a significant improvement in stutter in Forza Horizon 5
and Forza Horizon 4. (As games that had significant issues in sparse
binding related stutter).

Compared to the previous series [1], this specifically targets the VM
operations and keep everything else intact, including implicit sync on
kernel-initiated moves.

I've been able to pass a full Vulkan CTS run on Navi 10 with this.

Userspace code for this is available at [2] and a branch for the kernel
code is available at [3].

v2 changes:
- Drop the changes to flush split bindings eagerly as its incompatible
  with TLB flush quirks in current hardware. Drop the refactoring
  commits related to that change too.
- Fixed a missing doc warning.
- Removed an accidentally included ioctl change.

[1]: https://lore.kernel.org/all/20230821062005.109771-1-ishitatsuyuki@gmail.com/
[2]: https://gitlab.freedesktop.org/ishitatsuyuki/mesa/-/commits/vm-explicit-sync
[3]: https://github.com/ishitatsuyuki/linux/tree/explicit-sync-drm-misc-next

Tatsuyuki Ishi (3):
  drm/amdgpu: Don't implicit sync PRT maps.
  drm/amdgpu: Add flag to disable implicit sync for GEM operations.
  drm/amdgpu: Bump amdgpu driver version.

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 10 files changed, 73 insertions(+), 51 deletions(-)

-- 
2.42.0

[PATCH v2 1/3] drm/amdgpu: Don't implicit sync PRT maps.

[Tatsuyuki Ishi]

These are considered map operations rather than unmap, and there is no
point of doing implicit synchronization here.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index f5daadcec865..7b9762f1cddd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -902,7 +902,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & AMDGPU_PTE_VALID))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
-- 
2.42.0

[PATCH v2 1/3] drm/amdgpu: Don't implicit sync PRT maps.

[Tatsuyuki Ishi]

These are considered map operations rather than unmap, and there is no
point of doing implicit synchronization here.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index f5daadcec865..7b9762f1cddd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -902,7 +902,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & AMDGPU_PTE_VALID))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
-- 
2.42.0

[PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
Until now, the kernel applied the same rule as implicitly-synchronized
APIs like OpenGL, which with per-VM BOs made page table updates stall the
queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
drivers to opt-out of this behavior, while still ensuring adequate implicit
sync happens for kernel-initiated updates (e.g. BO moves).

We record whether to use implicit sync or not for each freed mapping. To
avoid increasing the mapping struct's size, this is union-ized with the
interval tree field which is unused after the unmap.

The reason this is done with a GEM ioctl flag, instead of being a VM /
context global setting, is that the current libdrm implementation shares
the DRM handle even between different kind of drivers (radeonsi vs radv).

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 9 files changed, 71 insertions(+), 50 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 7d6daf8d2bfa..10e129bff977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
 	struct amdgpu_device *adev = entry->adev;
 	struct amdgpu_vm *vm = bo_va->base.vm;
 
-	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
+	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
 
 	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 720011019741..612279e65bff 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 		}
 	}
 
-	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
+	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
 	if (r) {
 		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
 		goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index a1b15d0d6c48..cca68b89754e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
 		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
 		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
-		AMDGPU_VM_PAGE_NOALLOC;
+		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
 	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
-		AMDGPU_VM_PAGE_PRT;
+		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
 
 	struct drm_amdgpu_gem_va *args = data;
 	struct drm_gem_object *gobj;
@@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	uint64_t va_flags;
 	uint64_t vm_size;
+	bool sync_unmap;
 	int r = 0;
 
 	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
@@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
+
 	switch (args->operation) {
 	case AMDGPU_VA_OP_MAP:
 	case AMDGPU_VA_OP_UNMAP:
@@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 				     va_flags);
 		break;
 	case AMDGPU_VA_OP_UNMAP:
-		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
+		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
+				       sync_unmap);
 		break;
 
 	case AMDGPU_VA_OP_CLEAR:
 		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
 						args->va_address,
-						args->map_size);
+						args->map_size, sync_unmap);
 		break;
 	case AMDGPU_VA_OP_REPLACE:
 		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
 		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
 					     args->offset_in_bo, args->map_size,
-					     va_flags);
+					     va_flags, sync_unmap);
 		break;
 	default:
 		break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index f3ee83cdf97e..28be03f1bbcf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
 	struct rb_node			rb;
 	uint64_t			start;
 	uint64_t			last;
-	uint64_t			__subtree_last;
+	union {
+		/* BOs in interval tree only */
+		uint64_t		__subtree_last;
+		/* Freed BOs only */
+		bool			sync_unmap;
+	};
 	uint64_t			offset;
 	uint64_t			flags;
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 2fd1bfb35916..e71443c8c59b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			     __field(long, last)
 			     __field(u64, offset)
 			     __field(u64, flags)
+			     __field(bool, sync_unmap)
 			     ),
 
 	    TP_fast_assign(
@@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			   __entry->last = mapping->last;
 			   __entry->offset = mapping->offset;
 			   __entry->flags = mapping->flags;
+			   __entry->sync_unmap = mapping->sync_unmap;
 			   ),
-	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
+	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
 		      __entry->bo, __entry->start, __entry->last,
-		      __entry->offset, __entry->flags)
+		      __entry->offset, __entry->flags, __entry->sync_unmap)
 );
 
 DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 7b9762f1cddd..a74472e16952 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  * @immediate: immediate submission in a page fault
  * @unlocked: unlocked invalidation during MM callback
  * @flush_tlb: trigger tlb invalidation after update completed
+ * @sync_unmap: wait for BO users before unmapping
  * @resv: fences we need to sync to
  * @start: start of mapped range
  * @last: last mapped entry
@@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  */
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence)
 {
@@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
@@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 		trace_amdgpu_vm_bo_update(mapping);
 
 		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
-					   resv, mapping->start, mapping->last,
-					   update_flags, mapping->offset,
-					   vram_base, mem, pages_addr,
-					   last_update);
+					   true, resv, mapping->start,
+					   mapping->last, update_flags,
+					   mapping->offset, vram_base, mem,
+					   pages_addr, last_update);
 		if (r)
 			return r;
 	}
@@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 		    mapping->start < AMDGPU_GMC_HOLE_START)
 			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
 
-		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
+		r = amdgpu_vm_update_range(adev, vm, false, false, true,
+					   mapping->sync_unmap, resv,
 					   mapping->start, mapping->last,
 					   init_pte_value, 0, 0, NULL, NULL,
 					   &f);
@@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * @offset: requested offset in the BO
  * @size: BO size in bytes
  * @flags: attributes of pages (read/write/valid/etc.)
+ * @sync_unmap: wait for BO users before replacing existing mapping
  *
  * Add a mapping of the BO at the specefied addr into the VM. Replace existing
  * mappings as we do so.
@@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * Object has to be reserved and unreserved outside!
  */
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t saddr, uint64_t offset,
-			     uint64_t size, uint64_t flags)
+			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_bo *bo = bo_va->base.bo;
@@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
 	if (!mapping)
 		return -ENOMEM;
 
-	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
+	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
 	if (r) {
 		kfree(mapping);
 		return r;
@@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  * @adev: amdgpu_device pointer
  * @bo_va: bo_va to remove the address from
  * @saddr: where to the BO is mapped
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove a mapping of the BO at the specefied addr from the VM.
  *
@@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  *
  * Object has to be reserved and unreserved outside!
  */
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t saddr)
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t saddr, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_vm *vm = bo_va->base.vm;
@@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
 	list_del(&mapping->list);
 	amdgpu_vm_it_remove(mapping, &vm->va);
 	mapping->bo_va = NULL;
+	mapping->sync_unmap = sync_unmap;
 	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 
 	if (valid)
@@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * @vm: VM structure to use
  * @saddr: start of the range
  * @size: size of the range
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove all mappings in a range, split them as appropriate.
  *
@@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * 0 for success, error for failure.
  */
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size)
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
 	LIST_HEAD(removed);
@@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 		    tmp->last = eaddr;
 
 		tmp->bo_va = NULL;
+		tmp->sync_unmap = sync_unmap;
 		list_add(&tmp->list, &vm->freed);
 		trace_amdgpu_vm_bo_unmap(NULL, tmp);
 	}
@@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 		list_del(&mapping->list);
 		amdgpu_vm_it_remove(mapping, &vm->va);
 		mapping->bo_va = NULL;
+		mapping->sync_unmap = true;
 		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 		list_add(&mapping->list, &vm->freed);
 	}
@@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		goto error_unlock;
 	}
 
-	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
-				   addr, flags, value, 0, NULL, NULL, NULL);
+	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
+				   addr, addr, flags, value, 0, NULL, NULL,
+				   NULL);
 	if (r)
 		goto error_unlock;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 204ab13184ed..73b7b49fdb2e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence);
-int amdgpu_vm_bo_update(struct amdgpu_device *adev,
-			struct amdgpu_bo_va *bo_va,
+int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			bool clear);
 bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
 void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
@@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
 		     uint64_t addr, uint64_t offset,
 		     uint64_t size, uint64_t flags);
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t addr, uint64_t offset,
-			     uint64_t size, uint64_t flags);
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t addr);
+			     struct amdgpu_bo_va *bo_va, uint64_t addr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap);
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t addr, bool sync_unmap);
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size);
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap);
 struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
 							 uint64_t addr);
 void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index bb16b795d1bc..6eb4a0a4bc84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	pr_debug("[0x%llx 0x%llx]\n", start, last);
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
-				      fence);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
+				      start, last, init_pte_value, 0, 0, NULL,
+				      NULL, fence);
 }
 
 static int
@@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		 * different memory partition based on fpfn/lpfn, we should use
 		 * same vm_manager.vram_base_offset regardless memory partition.
 		 */
-		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
-					   last_start, prange->start + i,
-					   pte_flags,
-					   (last_start - prange->start) << PAGE_SHIFT,
-					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
-					   NULL, dma_addr, &vm->last_update);
+		r = amdgpu_vm_update_range(
+			adev, vm, false, false, flush_tlb, true, NULL,
+			last_start, prange->start + i, pte_flags,
+			(last_start - prange->start) << PAGE_SHIFT,
+			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
+			NULL, dma_addr, &vm->last_update);
 
 		for (j = last_start - prange->start; j <= i; j++)
 			dma_addr[j] |= last_domain;
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index f477eda6a2b8..3cdcc299956e 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
 #define AMDGPU_VM_MTYPE_RW		(5 << 5)
 /* don't allocate MALL */
 #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
+/* don't sync on unmap */
+#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
 
 struct drm_amdgpu_gem_va {
 	/** GEM object handle */
-- 
2.42.0

[PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

In Vulkan, it is the application's responsibility to perform adequate
synchronization before a sparse unmap, replace or BO destroy operation.
Until now, the kernel applied the same rule as implicitly-synchronized
APIs like OpenGL, which with per-VM BOs made page table updates stall the
queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
drivers to opt-out of this behavior, while still ensuring adequate implicit
sync happens for kernel-initiated updates (e.g. BO moves).

We record whether to use implicit sync or not for each freed mapping. To
avoid increasing the mapping struct's size, this is union-ized with the
interval tree field which is unused after the unmap.

The reason this is done with a GEM ioctl flag, instead of being a VM /
context global setting, is that the current libdrm implementation shares
the DRM handle even between different kind of drivers (radeonsi vs radv).

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
 include/uapi/drm/amdgpu_drm.h                 |  2 +
 9 files changed, 71 insertions(+), 50 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 7d6daf8d2bfa..10e129bff977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
 	struct amdgpu_device *adev = entry->adev;
 	struct amdgpu_vm *vm = bo_va->base.vm;
 
-	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
+	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
 
 	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 720011019741..612279e65bff 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 		}
 	}
 
-	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
+	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
 	if (r) {
 		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
 		goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index a1b15d0d6c48..cca68b89754e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
 		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
 		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
-		AMDGPU_VM_PAGE_NOALLOC;
+		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
 	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
-		AMDGPU_VM_PAGE_PRT;
+		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
 
 	struct drm_amdgpu_gem_va *args = data;
 	struct drm_gem_object *gobj;
@@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	uint64_t va_flags;
 	uint64_t vm_size;
+	bool sync_unmap;
 	int r = 0;
 
 	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
@@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
+
 	switch (args->operation) {
 	case AMDGPU_VA_OP_MAP:
 	case AMDGPU_VA_OP_UNMAP:
@@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
 				     va_flags);
 		break;
 	case AMDGPU_VA_OP_UNMAP:
-		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
+		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
+				       sync_unmap);
 		break;
 
 	case AMDGPU_VA_OP_CLEAR:
 		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
 						args->va_address,
-						args->map_size);
+						args->map_size, sync_unmap);
 		break;
 	case AMDGPU_VA_OP_REPLACE:
 		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
 		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
 					     args->offset_in_bo, args->map_size,
-					     va_flags);
+					     va_flags, sync_unmap);
 		break;
 	default:
 		break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index f3ee83cdf97e..28be03f1bbcf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
 	struct rb_node			rb;
 	uint64_t			start;
 	uint64_t			last;
-	uint64_t			__subtree_last;
+	union {
+		/* BOs in interval tree only */
+		uint64_t		__subtree_last;
+		/* Freed BOs only */
+		bool			sync_unmap;
+	};
 	uint64_t			offset;
 	uint64_t			flags;
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 2fd1bfb35916..e71443c8c59b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			     __field(long, last)
 			     __field(u64, offset)
 			     __field(u64, flags)
+			     __field(bool, sync_unmap)
 			     ),
 
 	    TP_fast_assign(
@@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
 			   __entry->last = mapping->last;
 			   __entry->offset = mapping->offset;
 			   __entry->flags = mapping->flags;
+			   __entry->sync_unmap = mapping->sync_unmap;
 			   ),
-	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
+	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
 		      __entry->bo, __entry->start, __entry->last,
-		      __entry->offset, __entry->flags)
+		      __entry->offset, __entry->flags, __entry->sync_unmap)
 );
 
 DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 7b9762f1cddd..a74472e16952 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  * @immediate: immediate submission in a page fault
  * @unlocked: unlocked invalidation during MM callback
  * @flush_tlb: trigger tlb invalidation after update completed
+ * @sync_unmap: wait for BO users before unmapping
  * @resv: fences we need to sync to
  * @start: start of mapped range
  * @last: last mapped entry
@@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
  */
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence)
 {
@@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	/* Implicitly sync to command submissions in the same VM before
 	 * unmapping. Sync to moving fences before mapping.
 	 */
-	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
+	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
 		sync_mode = AMDGPU_SYNC_EQ_OWNER;
 	else
 		sync_mode = AMDGPU_SYNC_EXPLICIT;
@@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 		trace_amdgpu_vm_bo_update(mapping);
 
 		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
-					   resv, mapping->start, mapping->last,
-					   update_flags, mapping->offset,
-					   vram_base, mem, pages_addr,
-					   last_update);
+					   true, resv, mapping->start,
+					   mapping->last, update_flags,
+					   mapping->offset, vram_base, mem,
+					   pages_addr, last_update);
 		if (r)
 			return r;
 	}
@@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 		    mapping->start < AMDGPU_GMC_HOLE_START)
 			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
 
-		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
+		r = amdgpu_vm_update_range(adev, vm, false, false, true,
+					   mapping->sync_unmap, resv,
 					   mapping->start, mapping->last,
 					   init_pte_value, 0, 0, NULL, NULL,
 					   &f);
@@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * @offset: requested offset in the BO
  * @size: BO size in bytes
  * @flags: attributes of pages (read/write/valid/etc.)
+ * @sync_unmap: wait for BO users before replacing existing mapping
  *
  * Add a mapping of the BO at the specefied addr into the VM. Replace existing
  * mappings as we do so.
@@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
  * Object has to be reserved and unreserved outside!
  */
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t saddr, uint64_t offset,
-			     uint64_t size, uint64_t flags)
+			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_bo *bo = bo_va->base.bo;
@@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
 	if (!mapping)
 		return -ENOMEM;
 
-	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
+	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
 	if (r) {
 		kfree(mapping);
 		return r;
@@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  * @adev: amdgpu_device pointer
  * @bo_va: bo_va to remove the address from
  * @saddr: where to the BO is mapped
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove a mapping of the BO at the specefied addr from the VM.
  *
@@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
  *
  * Object has to be reserved and unreserved outside!
  */
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t saddr)
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t saddr, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *mapping;
 	struct amdgpu_vm *vm = bo_va->base.vm;
@@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
 	list_del(&mapping->list);
 	amdgpu_vm_it_remove(mapping, &vm->va);
 	mapping->bo_va = NULL;
+	mapping->sync_unmap = sync_unmap;
 	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 
 	if (valid)
@@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * @vm: VM structure to use
  * @saddr: start of the range
  * @size: size of the range
+ * @sync_unmap: wait for BO users before unmapping
  *
  * Remove all mappings in a range, split them as appropriate.
  *
@@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
  * 0 for success, error for failure.
  */
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size)
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap)
 {
 	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
 	LIST_HEAD(removed);
@@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
 		    tmp->last = eaddr;
 
 		tmp->bo_va = NULL;
+		tmp->sync_unmap = sync_unmap;
 		list_add(&tmp->list, &vm->freed);
 		trace_amdgpu_vm_bo_unmap(NULL, tmp);
 	}
@@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
 		list_del(&mapping->list);
 		amdgpu_vm_it_remove(mapping, &vm->va);
 		mapping->bo_va = NULL;
+		mapping->sync_unmap = true;
 		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
 		list_add(&mapping->list, &vm->freed);
 	}
@@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		goto error_unlock;
 	}
 
-	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
-				   addr, flags, value, 0, NULL, NULL, NULL);
+	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
+				   addr, addr, flags, value, 0, NULL, NULL,
+				   NULL);
 	if (r)
 		goto error_unlock;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 204ab13184ed..73b7b49fdb2e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
 			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
 int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			   bool immediate, bool unlocked, bool flush_tlb,
-			   struct dma_resv *resv, uint64_t start, uint64_t last,
-			   uint64_t flags, uint64_t offset, uint64_t vram_base,
+			   bool sync_unmap, struct dma_resv *resv,
+			   uint64_t start, uint64_t last, uint64_t flags,
+			   uint64_t offset, uint64_t vram_base,
 			   struct ttm_resource *res, dma_addr_t *pages_addr,
 			   struct dma_fence **fence);
-int amdgpu_vm_bo_update(struct amdgpu_device *adev,
-			struct amdgpu_bo_va *bo_va,
+int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 			bool clear);
 bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
 void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
@@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
 		     uint64_t addr, uint64_t offset,
 		     uint64_t size, uint64_t flags);
 int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
-			     struct amdgpu_bo_va *bo_va,
-			     uint64_t addr, uint64_t offset,
-			     uint64_t size, uint64_t flags);
-int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
-		       struct amdgpu_bo_va *bo_va,
-		       uint64_t addr);
+			     struct amdgpu_bo_va *bo_va, uint64_t addr,
+			     uint64_t offset, uint64_t size, uint64_t flags,
+			     bool sync_unmap);
+int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
+		       uint64_t addr, bool sync_unmap);
 int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
-				struct amdgpu_vm *vm,
-				uint64_t saddr, uint64_t size);
+				struct amdgpu_vm *vm, uint64_t saddr,
+				uint64_t size, bool sync_unmap);
 struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
 							 uint64_t addr);
 void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index bb16b795d1bc..6eb4a0a4bc84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	pr_debug("[0x%llx 0x%llx]\n", start, last);
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
-				      fence);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
+				      start, last, init_pte_value, 0, 0, NULL,
+				      NULL, fence);
 }
 
 static int
@@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		 * different memory partition based on fpfn/lpfn, we should use
 		 * same vm_manager.vram_base_offset regardless memory partition.
 		 */
-		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
-					   last_start, prange->start + i,
-					   pte_flags,
-					   (last_start - prange->start) << PAGE_SHIFT,
-					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
-					   NULL, dma_addr, &vm->last_update);
+		r = amdgpu_vm_update_range(
+			adev, vm, false, false, flush_tlb, true, NULL,
+			last_start, prange->start + i, pte_flags,
+			(last_start - prange->start) << PAGE_SHIFT,
+			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
+			NULL, dma_addr, &vm->last_update);
 
 		for (j = last_start - prange->start; j <= i; j++)
 			dma_addr[j] |= last_domain;
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index f477eda6a2b8..3cdcc299956e 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
 #define AMDGPU_VM_MTYPE_RW		(5 << 5)
 /* don't allocate MALL */
 #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
+/* don't sync on unmap */
+#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
 
 struct drm_amdgpu_gem_va {
 	/** GEM object handle */
-- 
2.42.0

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

  Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
> In Vulkan, it is the application's responsibility to perform adequate
> synchronization before a sparse unmap, replace or BO destroy operation.
> Until now, the kernel applied the same rule as implicitly-synchronized
> APIs like OpenGL, which with per-VM BOs made page table updates stall the
> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> drivers to opt-out of this behavior, while still ensuring adequate implicit
> sync happens for kernel-initiated updates (e.g. BO moves).
>
> We record whether to use implicit sync or not for each freed mapping. To
> avoid increasing the mapping struct's size, this is union-ized with the
> interval tree field which is unused after the unmap.
>
> The reason this is done with a GEM ioctl flag, instead of being a VM /
> context global setting, is that the current libdrm implementation shares
> the DRM handle even between different kind of drivers (radeonsi vs radv).

It would be nice if we could make this more future prove by not using a 
flag, but rather a drm_syncobj.

You can extend the drm_amdgpu_gem_va structure by adding a drm_syncobj 
handle and timeline point at the end.

If the syncobj/timeline point results in a fence we give that as input 
dependency the operation has to wait for.

And output fence can come later on as well, but that one is much more 
harder to handle.

Regards,
Christian.

>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>   drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>   include/uapi/drm/amdgpu_drm.h                 |  2 +
>   9 files changed, 71 insertions(+), 50 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index 7d6daf8d2bfa..10e129bff977 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
>   	struct amdgpu_device *adev = entry->adev;
>   	struct amdgpu_vm *vm = bo_va->base.vm;
>   
> -	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
> +	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>   
>   	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> index 720011019741..612279e65bff 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   		}
>   	}
>   
> -	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
> +	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>   	if (r) {
>   		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>   		goto error;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index a1b15d0d6c48..cca68b89754e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>   		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>   		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
> -		AMDGPU_VM_PAGE_NOALLOC;
> +		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>   	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
> -		AMDGPU_VM_PAGE_PRT;
> +		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>   
>   	struct drm_amdgpu_gem_va *args = data;
>   	struct drm_gem_object *gobj;
> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   	struct drm_exec exec;
>   	uint64_t va_flags;
>   	uint64_t vm_size;
> +	bool sync_unmap;
>   	int r = 0;
>   
>   	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   		return -EINVAL;
>   	}
>   
> +	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
> +
>   	switch (args->operation) {
>   	case AMDGPU_VA_OP_MAP:
>   	case AMDGPU_VA_OP_UNMAP:
> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   				     va_flags);
>   		break;
>   	case AMDGPU_VA_OP_UNMAP:
> -		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
> +		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
> +				       sync_unmap);
>   		break;
>   
>   	case AMDGPU_VA_OP_CLEAR:
>   		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>   						args->va_address,
> -						args->map_size);
> +						args->map_size, sync_unmap);
>   		break;
>   	case AMDGPU_VA_OP_REPLACE:
>   		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>   		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>   					     args->offset_in_bo, args->map_size,
> -					     va_flags);
> +					     va_flags, sync_unmap);
>   		break;
>   	default:
>   		break;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> index f3ee83cdf97e..28be03f1bbcf 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>   	struct rb_node			rb;
>   	uint64_t			start;
>   	uint64_t			last;
> -	uint64_t			__subtree_last;
> +	union {
> +		/* BOs in interval tree only */
> +		uint64_t		__subtree_last;
> +		/* Freed BOs only */
> +		bool			sync_unmap;
> +	};
>   	uint64_t			offset;
>   	uint64_t			flags;
>   };
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> index 2fd1bfb35916..e71443c8c59b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>   			     __field(long, last)
>   			     __field(u64, offset)
>   			     __field(u64, flags)
> +			     __field(bool, sync_unmap)
>   			     ),
>   
>   	    TP_fast_assign(
> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>   			   __entry->last = mapping->last;
>   			   __entry->offset = mapping->offset;
>   			   __entry->flags = mapping->flags;
> +			   __entry->sync_unmap = mapping->sync_unmap;
>   			   ),
> -	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
> +	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
>   		      __entry->bo, __entry->start, __entry->last,
> -		      __entry->offset, __entry->flags)
> +		      __entry->offset, __entry->flags, __entry->sync_unmap)
>   );
>   
>   DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 7b9762f1cddd..a74472e16952 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>    * @immediate: immediate submission in a page fault
>    * @unlocked: unlocked invalidation during MM callback
>    * @flush_tlb: trigger tlb invalidation after update completed
> + * @sync_unmap: wait for BO users before unmapping
>    * @resv: fences we need to sync to
>    * @start: start of mapped range
>    * @last: last mapped entry
> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>    */
>   int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   			   bool immediate, bool unlocked, bool flush_tlb,
> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
> +			   bool sync_unmap, struct dma_resv *resv,
> +			   uint64_t start, uint64_t last, uint64_t flags,
> +			   uint64_t offset, uint64_t vram_base,
>   			   struct ttm_resource *res, dma_addr_t *pages_addr,
>   			   struct dma_fence **fence)
>   {
> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   	/* Implicitly sync to command submissions in the same VM before
>   	 * unmapping. Sync to moving fences before mapping.
>   	 */
> -	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
> +	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>   		sync_mode = AMDGPU_SYNC_EQ_OWNER;
>   	else
>   		sync_mode = AMDGPU_SYNC_EXPLICIT;
> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   		trace_amdgpu_vm_bo_update(mapping);
>   
>   		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
> -					   resv, mapping->start, mapping->last,
> -					   update_flags, mapping->offset,
> -					   vram_base, mem, pages_addr,
> -					   last_update);
> +					   true, resv, mapping->start,
> +					   mapping->last, update_flags,
> +					   mapping->offset, vram_base, mem,
> +					   pages_addr, last_update);
>   		if (r)
>   			return r;
>   	}
> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
>   		    mapping->start < AMDGPU_GMC_HOLE_START)
>   			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>   
> -		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
> +		r = amdgpu_vm_update_range(adev, vm, false, false, true,
> +					   mapping->sync_unmap, resv,
>   					   mapping->start, mapping->last,
>   					   init_pte_value, 0, 0, NULL, NULL,
>   					   &f);
> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>    * @offset: requested offset in the BO
>    * @size: BO size in bytes
>    * @flags: attributes of pages (read/write/valid/etc.)
> + * @sync_unmap: wait for BO users before replacing existing mapping
>    *
>    * Add a mapping of the BO at the specefied addr into the VM. Replace existing
>    * mappings as we do so.
> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>    * Object has to be reserved and unreserved outside!
>    */
>   int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
> -			     struct amdgpu_bo_va *bo_va,
> -			     uint64_t saddr, uint64_t offset,
> -			     uint64_t size, uint64_t flags)
> +			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
> +			     uint64_t offset, uint64_t size, uint64_t flags,
> +			     bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *mapping;
>   	struct amdgpu_bo *bo = bo_va->base.bo;
> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>   	if (!mapping)
>   		return -ENOMEM;
>   
> -	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
> +	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
>   	if (r) {
>   		kfree(mapping);
>   		return r;
> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>    * @adev: amdgpu_device pointer
>    * @bo_va: bo_va to remove the address from
>    * @saddr: where to the BO is mapped
> + * @sync_unmap: wait for BO users before unmapping
>    *
>    * Remove a mapping of the BO at the specefied addr from the VM.
>    *
> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>    *
>    * Object has to be reserved and unreserved outside!
>    */
> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
> -		       struct amdgpu_bo_va *bo_va,
> -		       uint64_t saddr)
> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
> +		       uint64_t saddr, bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *mapping;
>   	struct amdgpu_vm *vm = bo_va->base.vm;
> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>   	list_del(&mapping->list);
>   	amdgpu_vm_it_remove(mapping, &vm->va);
>   	mapping->bo_va = NULL;
> +	mapping->sync_unmap = sync_unmap;
>   	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>   
>   	if (valid)
> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>    * @vm: VM structure to use
>    * @saddr: start of the range
>    * @size: size of the range
> + * @sync_unmap: wait for BO users before unmapping
>    *
>    * Remove all mappings in a range, split them as appropriate.
>    *
> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>    * 0 for success, error for failure.
>    */
>   int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
> -				struct amdgpu_vm *vm,
> -				uint64_t saddr, uint64_t size)
> +				struct amdgpu_vm *vm, uint64_t saddr,
> +				uint64_t size, bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>   	LIST_HEAD(removed);
> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   		    tmp->last = eaddr;
>   
>   		tmp->bo_va = NULL;
> +		tmp->sync_unmap = sync_unmap;
>   		list_add(&tmp->list, &vm->freed);
>   		trace_amdgpu_vm_bo_unmap(NULL, tmp);
>   	}
> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>   		list_del(&mapping->list);
>   		amdgpu_vm_it_remove(mapping, &vm->va);
>   		mapping->bo_va = NULL;
> +		mapping->sync_unmap = true;
>   		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>   		list_add(&mapping->list, &vm->freed);
>   	}
> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
>   		goto error_unlock;
>   	}
>   
> -	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
> -				   addr, flags, value, 0, NULL, NULL, NULL);
> +	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
> +				   addr, addr, flags, value, 0, NULL, NULL,
> +				   NULL);
>   	if (r)
>   		goto error_unlock;
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 204ab13184ed..73b7b49fdb2e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>   int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   			   bool immediate, bool unlocked, bool flush_tlb,
> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
> +			   bool sync_unmap, struct dma_resv *resv,
> +			   uint64_t start, uint64_t last, uint64_t flags,
> +			   uint64_t offset, uint64_t vram_base,
>   			   struct ttm_resource *res, dma_addr_t *pages_addr,
>   			   struct dma_fence **fence);
> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
> -			struct amdgpu_bo_va *bo_va,
> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   			bool clear);
>   bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>   void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>   		     uint64_t addr, uint64_t offset,
>   		     uint64_t size, uint64_t flags);
>   int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
> -			     struct amdgpu_bo_va *bo_va,
> -			     uint64_t addr, uint64_t offset,
> -			     uint64_t size, uint64_t flags);
> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
> -		       struct amdgpu_bo_va *bo_va,
> -		       uint64_t addr);
> +			     struct amdgpu_bo_va *bo_va, uint64_t addr,
> +			     uint64_t offset, uint64_t size, uint64_t flags,
> +			     bool sync_unmap);
> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
> +		       uint64_t addr, bool sync_unmap);
>   int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
> -				struct amdgpu_vm *vm,
> -				uint64_t saddr, uint64_t size);
> +				struct amdgpu_vm *vm, uint64_t saddr,
> +				uint64_t size, bool sync_unmap);
>   struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
>   							 uint64_t addr);
>   void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> index bb16b795d1bc..6eb4a0a4bc84 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   
>   	pr_debug("[0x%llx 0x%llx]\n", start, last);
>   
> -	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
> -				      last, init_pte_value, 0, 0, NULL, NULL,
> -				      fence);
> +	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
> +				      start, last, init_pte_value, 0, 0, NULL,
> +				      NULL, fence);
>   }
>   
>   static int
> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
>   		 * different memory partition based on fpfn/lpfn, we should use
>   		 * same vm_manager.vram_base_offset regardless memory partition.
>   		 */
> -		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
> -					   last_start, prange->start + i,
> -					   pte_flags,
> -					   (last_start - prange->start) << PAGE_SHIFT,
> -					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
> -					   NULL, dma_addr, &vm->last_update);
> +		r = amdgpu_vm_update_range(
> +			adev, vm, false, false, flush_tlb, true, NULL,
> +			last_start, prange->start + i, pte_flags,
> +			(last_start - prange->start) << PAGE_SHIFT,
> +			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
> +			NULL, dma_addr, &vm->last_update);
>   
>   		for (j = last_start - prange->start; j <= i; j++)
>   			dma_addr[j] |= last_domain;
> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
> index f477eda6a2b8..3cdcc299956e 100644
> --- a/include/uapi/drm/amdgpu_drm.h
> +++ b/include/uapi/drm/amdgpu_drm.h
> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>   #define AMDGPU_VM_MTYPE_RW		(5 << 5)
>   /* don't allocate MALL */
>   #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
> +/* don't sync on unmap */
> +#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
>   
>   struct drm_amdgpu_gem_va {
>   	/** GEM object handle */

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

  Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
> In Vulkan, it is the application's responsibility to perform adequate
> synchronization before a sparse unmap, replace or BO destroy operation.
> Until now, the kernel applied the same rule as implicitly-synchronized
> APIs like OpenGL, which with per-VM BOs made page table updates stall the
> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
> drivers to opt-out of this behavior, while still ensuring adequate implicit
> sync happens for kernel-initiated updates (e.g. BO moves).
>
> We record whether to use implicit sync or not for each freed mapping. To
> avoid increasing the mapping struct's size, this is union-ized with the
> interval tree field which is unused after the unmap.
>
> The reason this is done with a GEM ioctl flag, instead of being a VM /
> context global setting, is that the current libdrm implementation shares
> the DRM handle even between different kind of drivers (radeonsi vs radv).

It would be nice if we could make this more future prove by not using a 
flag, but rather a drm_syncobj.

You can extend the drm_amdgpu_gem_va structure by adding a drm_syncobj 
handle and timeline point at the end.

If the syncobj/timeline point results in a fence we give that as input 
dependency the operation has to wait for.

And output fence can come later on as well, but that one is much more 
harder to handle.

Regards,
Christian.

>
> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
> ---
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>   drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>   include/uapi/drm/amdgpu_drm.h                 |  2 +
>   9 files changed, 71 insertions(+), 50 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index 7d6daf8d2bfa..10e129bff977 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
>   	struct amdgpu_device *adev = entry->adev;
>   	struct amdgpu_vm *vm = bo_va->base.vm;
>   
> -	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
> +	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>   
>   	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> index 720011019741..612279e65bff 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   		}
>   	}
>   
> -	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
> +	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>   	if (r) {
>   		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>   		goto error;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index a1b15d0d6c48..cca68b89754e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>   		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>   		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
> -		AMDGPU_VM_PAGE_NOALLOC;
> +		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>   	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
> -		AMDGPU_VM_PAGE_PRT;
> +		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>   
>   	struct drm_amdgpu_gem_va *args = data;
>   	struct drm_gem_object *gobj;
> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   	struct drm_exec exec;
>   	uint64_t va_flags;
>   	uint64_t vm_size;
> +	bool sync_unmap;
>   	int r = 0;
>   
>   	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   		return -EINVAL;
>   	}
>   
> +	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
> +
>   	switch (args->operation) {
>   	case AMDGPU_VA_OP_MAP:
>   	case AMDGPU_VA_OP_UNMAP:
> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>   				     va_flags);
>   		break;
>   	case AMDGPU_VA_OP_UNMAP:
> -		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
> +		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
> +				       sync_unmap);
>   		break;
>   
>   	case AMDGPU_VA_OP_CLEAR:
>   		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>   						args->va_address,
> -						args->map_size);
> +						args->map_size, sync_unmap);
>   		break;
>   	case AMDGPU_VA_OP_REPLACE:
>   		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>   		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>   					     args->offset_in_bo, args->map_size,
> -					     va_flags);
> +					     va_flags, sync_unmap);
>   		break;
>   	default:
>   		break;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> index f3ee83cdf97e..28be03f1bbcf 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>   	struct rb_node			rb;
>   	uint64_t			start;
>   	uint64_t			last;
> -	uint64_t			__subtree_last;
> +	union {
> +		/* BOs in interval tree only */
> +		uint64_t		__subtree_last;
> +		/* Freed BOs only */
> +		bool			sync_unmap;
> +	};
>   	uint64_t			offset;
>   	uint64_t			flags;
>   };
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> index 2fd1bfb35916..e71443c8c59b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>   			     __field(long, last)
>   			     __field(u64, offset)
>   			     __field(u64, flags)
> +			     __field(bool, sync_unmap)
>   			     ),
>   
>   	    TP_fast_assign(
> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>   			   __entry->last = mapping->last;
>   			   __entry->offset = mapping->offset;
>   			   __entry->flags = mapping->flags;
> +			   __entry->sync_unmap = mapping->sync_unmap;
>   			   ),
> -	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
> +	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
>   		      __entry->bo, __entry->start, __entry->last,
> -		      __entry->offset, __entry->flags)
> +		      __entry->offset, __entry->flags, __entry->sync_unmap)
>   );
>   
>   DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 7b9762f1cddd..a74472e16952 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>    * @immediate: immediate submission in a page fault
>    * @unlocked: unlocked invalidation during MM callback
>    * @flush_tlb: trigger tlb invalidation after update completed
> + * @sync_unmap: wait for BO users before unmapping
>    * @resv: fences we need to sync to
>    * @start: start of mapped range
>    * @last: last mapped entry
> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>    */
>   int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   			   bool immediate, bool unlocked, bool flush_tlb,
> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
> +			   bool sync_unmap, struct dma_resv *resv,
> +			   uint64_t start, uint64_t last, uint64_t flags,
> +			   uint64_t offset, uint64_t vram_base,
>   			   struct ttm_resource *res, dma_addr_t *pages_addr,
>   			   struct dma_fence **fence)
>   {
> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   	/* Implicitly sync to command submissions in the same VM before
>   	 * unmapping. Sync to moving fences before mapping.
>   	 */
> -	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
> +	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>   		sync_mode = AMDGPU_SYNC_EQ_OWNER;
>   	else
>   		sync_mode = AMDGPU_SYNC_EXPLICIT;
> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   		trace_amdgpu_vm_bo_update(mapping);
>   
>   		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
> -					   resv, mapping->start, mapping->last,
> -					   update_flags, mapping->offset,
> -					   vram_base, mem, pages_addr,
> -					   last_update);
> +					   true, resv, mapping->start,
> +					   mapping->last, update_flags,
> +					   mapping->offset, vram_base, mem,
> +					   pages_addr, last_update);
>   		if (r)
>   			return r;
>   	}
> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
>   		    mapping->start < AMDGPU_GMC_HOLE_START)
>   			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>   
> -		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
> +		r = amdgpu_vm_update_range(adev, vm, false, false, true,
> +					   mapping->sync_unmap, resv,
>   					   mapping->start, mapping->last,
>   					   init_pte_value, 0, 0, NULL, NULL,
>   					   &f);
> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>    * @offset: requested offset in the BO
>    * @size: BO size in bytes
>    * @flags: attributes of pages (read/write/valid/etc.)
> + * @sync_unmap: wait for BO users before replacing existing mapping
>    *
>    * Add a mapping of the BO at the specefied addr into the VM. Replace existing
>    * mappings as we do so.
> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>    * Object has to be reserved and unreserved outside!
>    */
>   int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
> -			     struct amdgpu_bo_va *bo_va,
> -			     uint64_t saddr, uint64_t offset,
> -			     uint64_t size, uint64_t flags)
> +			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
> +			     uint64_t offset, uint64_t size, uint64_t flags,
> +			     bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *mapping;
>   	struct amdgpu_bo *bo = bo_va->base.bo;
> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>   	if (!mapping)
>   		return -ENOMEM;
>   
> -	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
> +	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
>   	if (r) {
>   		kfree(mapping);
>   		return r;
> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>    * @adev: amdgpu_device pointer
>    * @bo_va: bo_va to remove the address from
>    * @saddr: where to the BO is mapped
> + * @sync_unmap: wait for BO users before unmapping
>    *
>    * Remove a mapping of the BO at the specefied addr from the VM.
>    *
> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>    *
>    * Object has to be reserved and unreserved outside!
>    */
> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
> -		       struct amdgpu_bo_va *bo_va,
> -		       uint64_t saddr)
> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
> +		       uint64_t saddr, bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *mapping;
>   	struct amdgpu_vm *vm = bo_va->base.vm;
> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>   	list_del(&mapping->list);
>   	amdgpu_vm_it_remove(mapping, &vm->va);
>   	mapping->bo_va = NULL;
> +	mapping->sync_unmap = sync_unmap;
>   	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>   
>   	if (valid)
> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>    * @vm: VM structure to use
>    * @saddr: start of the range
>    * @size: size of the range
> + * @sync_unmap: wait for BO users before unmapping
>    *
>    * Remove all mappings in a range, split them as appropriate.
>    *
> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>    * 0 for success, error for failure.
>    */
>   int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
> -				struct amdgpu_vm *vm,
> -				uint64_t saddr, uint64_t size)
> +				struct amdgpu_vm *vm, uint64_t saddr,
> +				uint64_t size, bool sync_unmap)
>   {
>   	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>   	LIST_HEAD(removed);
> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>   		    tmp->last = eaddr;
>   
>   		tmp->bo_va = NULL;
> +		tmp->sync_unmap = sync_unmap;
>   		list_add(&tmp->list, &vm->freed);
>   		trace_amdgpu_vm_bo_unmap(NULL, tmp);
>   	}
> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>   		list_del(&mapping->list);
>   		amdgpu_vm_it_remove(mapping, &vm->va);
>   		mapping->bo_va = NULL;
> +		mapping->sync_unmap = true;
>   		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>   		list_add(&mapping->list, &vm->freed);
>   	}
> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
>   		goto error_unlock;
>   	}
>   
> -	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
> -				   addr, flags, value, 0, NULL, NULL, NULL);
> +	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
> +				   addr, addr, flags, value, 0, NULL, NULL,
> +				   NULL);
>   	if (r)
>   		goto error_unlock;
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 204ab13184ed..73b7b49fdb2e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>   			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>   int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   			   bool immediate, bool unlocked, bool flush_tlb,
> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
> +			   bool sync_unmap, struct dma_resv *resv,
> +			   uint64_t start, uint64_t last, uint64_t flags,
> +			   uint64_t offset, uint64_t vram_base,
>   			   struct ttm_resource *res, dma_addr_t *pages_addr,
>   			   struct dma_fence **fence);
> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
> -			struct amdgpu_bo_va *bo_va,
> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>   			bool clear);
>   bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>   void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>   		     uint64_t addr, uint64_t offset,
>   		     uint64_t size, uint64_t flags);
>   int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
> -			     struct amdgpu_bo_va *bo_va,
> -			     uint64_t addr, uint64_t offset,
> -			     uint64_t size, uint64_t flags);
> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
> -		       struct amdgpu_bo_va *bo_va,
> -		       uint64_t addr);
> +			     struct amdgpu_bo_va *bo_va, uint64_t addr,
> +			     uint64_t offset, uint64_t size, uint64_t flags,
> +			     bool sync_unmap);
> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
> +		       uint64_t addr, bool sync_unmap);
>   int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
> -				struct amdgpu_vm *vm,
> -				uint64_t saddr, uint64_t size);
> +				struct amdgpu_vm *vm, uint64_t saddr,
> +				uint64_t size, bool sync_unmap);
>   struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
>   							 uint64_t addr);
>   void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> index bb16b795d1bc..6eb4a0a4bc84 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>   
>   	pr_debug("[0x%llx 0x%llx]\n", start, last);
>   
> -	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
> -				      last, init_pte_value, 0, 0, NULL, NULL,
> -				      fence);
> +	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
> +				      start, last, init_pte_value, 0, 0, NULL,
> +				      NULL, fence);
>   }
>   
>   static int
> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
>   		 * different memory partition based on fpfn/lpfn, we should use
>   		 * same vm_manager.vram_base_offset regardless memory partition.
>   		 */
> -		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
> -					   last_start, prange->start + i,
> -					   pte_flags,
> -					   (last_start - prange->start) << PAGE_SHIFT,
> -					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
> -					   NULL, dma_addr, &vm->last_update);
> +		r = amdgpu_vm_update_range(
> +			adev, vm, false, false, flush_tlb, true, NULL,
> +			last_start, prange->start + i, pte_flags,
> +			(last_start - prange->start) << PAGE_SHIFT,
> +			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
> +			NULL, dma_addr, &vm->last_update);
>   
>   		for (j = last_start - prange->start; j <= i; j++)
>   			dma_addr[j] |= last_domain;
> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
> index f477eda6a2b8..3cdcc299956e 100644
> --- a/include/uapi/drm/amdgpu_drm.h
> +++ b/include/uapi/drm/amdgpu_drm.h
> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>   #define AMDGPU_VM_MTYPE_RW		(5 << 5)
>   /* don't allocate MALL */
>   #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
> +/* don't sync on unmap */
> +#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
>   
>   struct drm_amdgpu_gem_va {
>   	/** GEM object handle */

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

[-- Attachment #1: Type: text/plain, Size: 20157 bytes --]

> On Nov 6, 2023, at 22:44, Christian König <christian.koenig@amd.com> wrote:
> 
>  Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
>> In Vulkan, it is the application's responsibility to perform adequate
>> synchronization before a sparse unmap, replace or BO destroy operation.
>> Until now, the kernel applied the same rule as implicitly-synchronized
>> APIs like OpenGL, which with per-VM BOs made page table updates stall the
>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>> drivers to opt-out of this behavior, while still ensuring adequate implicit
>> sync happens for kernel-initiated updates (e.g. BO moves).
>> 
>> We record whether to use implicit sync or not for each freed mapping. To
>> avoid increasing the mapping struct's size, this is union-ized with the
>> interval tree field which is unused after the unmap.
>> 
>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>> context global setting, is that the current libdrm implementation shares
>> the DRM handle even between different kind of drivers (radeonsi vs radv).
> 
> It would be nice if we could make this more future prove by not using a flag, but rather a drm_syncobj.

There is asynchronous VM_BIND and synchronous VM_BIND. Using syncobjs address asynchronous binds, but what this patch set solves is to add an explicitly synced synchronous bind.

Even within Vulkan, there are use cases for synchronous binds. This is when a non-sparse BO is destroyed (or created but that’s not synchronized). Such operations should still be explicit sync, unlike OpenGL where it syncs to previous submissions. So adding asynchronous bind doesn’t supersede this need.

I’ve also thought whether we can just make the unmap asynchronous, since the spec requires that destroyed stuff are not accessed in any way, but I think it will complicate behavior when the destruction of BO immediately follows.

We should implement asynchronous bind someday to make vkQueueBindSparse work (even) better, but that will likely involve a larger scope including the scheduler. Getting synchronous but explicitly synced binds should be simpler and a good incremental step.

Tatsuyuki.

> You can extend the drm_amdgpu_gem_va structure by adding a drm_syncobj handle and timeline point at the end.
> 
> If the syncobj/timeline point results in a fence we give that as input dependency the operation has to wait for.
> 
> And output fence can come later on as well, but that one is much more harder to handle.
> 
> Regards,
> Christian.
> 
>> 
>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> ---
>>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>>  drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>>  include/uapi/drm/amdgpu_drm.h                 |  2 +
>>  9 files changed, 71 insertions(+), 50 deletions(-)
>> 
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> index 7d6daf8d2bfa..10e129bff977 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
>>  	struct amdgpu_device *adev = entry->adev;
>>  	struct amdgpu_vm *vm = bo_va->base.vm;
>>  -	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
>> +	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>>    	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> index 720011019741..612279e65bff 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  		}
>>  	}
>>  -	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
>> +	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>>  	if (r) {
>>  		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>>  		goto error;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> index a1b15d0d6c48..cca68b89754e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>>  		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>>  		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
>> -		AMDGPU_VM_PAGE_NOALLOC;
>> +		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>>  	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
>> -		AMDGPU_VM_PAGE_PRT;
>> +		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>>    	struct drm_amdgpu_gem_va *args = data;
>>  	struct drm_gem_object *gobj;
>> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  	struct drm_exec exec;
>>  	uint64_t va_flags;
>>  	uint64_t vm_size;
>> +	bool sync_unmap;
>>  	int r = 0;
>>    	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
>> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  		return -EINVAL;
>>  	}
>>  +	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
>> +
>>  	switch (args->operation) {
>>  	case AMDGPU_VA_OP_MAP:
>>  	case AMDGPU_VA_OP_UNMAP:
>> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  				     va_flags);
>>  		break;
>>  	case AMDGPU_VA_OP_UNMAP:
>> -		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
>> +		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
>> +				       sync_unmap);
>>  		break;
>>    	case AMDGPU_VA_OP_CLEAR:
>>  		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>>  						args->va_address,
>> -						args->map_size);
>> +						args->map_size, sync_unmap);
>>  		break;
>>  	case AMDGPU_VA_OP_REPLACE:
>>  		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>>  		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>>  					     args->offset_in_bo, args->map_size,
>> -					     va_flags);
>> +					     va_flags, sync_unmap);
>>  		break;
>>  	default:
>>  		break;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> index f3ee83cdf97e..28be03f1bbcf 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>>  	struct rb_node			rb;
>>  	uint64_t			start;
>>  	uint64_t			last;
>> -	uint64_t			__subtree_last;
>> +	union {
>> +		/* BOs in interval tree only */
>> +		uint64_t		__subtree_last;
>> +		/* Freed BOs only */
>> +		bool			sync_unmap;
>> +	};
>>  	uint64_t			offset;
>>  	uint64_t			flags;
>>  };
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> index 2fd1bfb35916..e71443c8c59b 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>  			     __field(long, last)
>>  			     __field(u64, offset)
>>  			     __field(u64, flags)
>> +			     __field(bool, sync_unmap)
>>  			     ),
>>    	    TP_fast_assign(
>> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>  			   __entry->last = mapping->last;
>>  			   __entry->offset = mapping->offset;
>>  			   __entry->flags = mapping->flags;
>> +			   __entry->sync_unmap = mapping->sync_unmap;
>>  			   ),
>> -	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
>> +	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
>>  		      __entry->bo, __entry->start, __entry->last,
>> -		      __entry->offset, __entry->flags)
>> +		      __entry->offset, __entry->flags, __entry->sync_unmap)
>>  );
>>    DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> index 7b9762f1cddd..a74472e16952 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>>   * @immediate: immediate submission in a page fault
>>   * @unlocked: unlocked invalidation during MM callback
>>   * @flush_tlb: trigger tlb invalidation after update completed
>> + * @sync_unmap: wait for BO users before unmapping
>>   * @resv: fences we need to sync to
>>   * @start: start of mapped range
>>   * @last: last mapped entry
>> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>>   */
>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  			   bool immediate, bool unlocked, bool flush_tlb,
>> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
>> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
>> +			   bool sync_unmap, struct dma_resv *resv,
>> +			   uint64_t start, uint64_t last, uint64_t flags,
>> +			   uint64_t offset, uint64_t vram_base,
>>  			   struct ttm_resource *res, dma_addr_t *pages_addr,
>>  			   struct dma_fence **fence)
>>  {
>> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  	/* Implicitly sync to command submissions in the same VM before
>>  	 * unmapping. Sync to moving fences before mapping.
>>  	 */
>> -	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
>> +	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>>  		sync_mode = AMDGPU_SYNC_EQ_OWNER;
>>  	else
>>  		sync_mode = AMDGPU_SYNC_EXPLICIT;
>> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  		trace_amdgpu_vm_bo_update(mapping);
>>    		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
>> -					   resv, mapping->start, mapping->last,
>> -					   update_flags, mapping->offset,
>> -					   vram_base, mem, pages_addr,
>> -					   last_update);
>> +					   true, resv, mapping->start,
>> +					   mapping->last, update_flags,
>> +					   mapping->offset, vram_base, mem,
>> +					   pages_addr, last_update);
>>  		if (r)
>>  			return r;
>>  	}
>> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
>>  		    mapping->start < AMDGPU_GMC_HOLE_START)
>>  			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>>  -		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
>> +		r = amdgpu_vm_update_range(adev, vm, false, false, true,
>> +					   mapping->sync_unmap, resv,
>>  					   mapping->start, mapping->last,
>>  					   init_pte_value, 0, 0, NULL, NULL,
>>  					   &f);
>> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>   * @offset: requested offset in the BO
>>   * @size: BO size in bytes
>>   * @flags: attributes of pages (read/write/valid/etc.)
>> + * @sync_unmap: wait for BO users before replacing existing mapping
>>   *
>>   * Add a mapping of the BO at the specefied addr into the VM. Replace existing
>>   * mappings as we do so.
>> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>   * Object has to be reserved and unreserved outside!
>>   */
>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>> -			     struct amdgpu_bo_va *bo_va,
>> -			     uint64_t saddr, uint64_t offset,
>> -			     uint64_t size, uint64_t flags)
>> +			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
>> +			     uint64_t offset, uint64_t size, uint64_t flags,
>> +			     bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *mapping;
>>  	struct amdgpu_bo *bo = bo_va->base.bo;
>> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>  	if (!mapping)
>>  		return -ENOMEM;
>>  -	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
>> +	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
>>  	if (r) {
>>  		kfree(mapping);
>>  		return r;
>> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>   * @adev: amdgpu_device pointer
>>   * @bo_va: bo_va to remove the address from
>>   * @saddr: where to the BO is mapped
>> + * @sync_unmap: wait for BO users before unmapping
>>   *
>>   * Remove a mapping of the BO at the specefied addr from the VM.
>>   *
>> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>   *
>>   * Object has to be reserved and unreserved outside!
>>   */
>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>> -		       struct amdgpu_bo_va *bo_va,
>> -		       uint64_t saddr)
>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>> +		       uint64_t saddr, bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *mapping;
>>  	struct amdgpu_vm *vm = bo_va->base.vm;
>> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>  	list_del(&mapping->list);
>>  	amdgpu_vm_it_remove(mapping, &vm->va);
>>  	mapping->bo_va = NULL;
>> +	mapping->sync_unmap = sync_unmap;
>>  	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>    	if (valid)
>> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>   * @vm: VM structure to use
>>   * @saddr: start of the range
>>   * @size: size of the range
>> + * @sync_unmap: wait for BO users before unmapping
>>   *
>>   * Remove all mappings in a range, split them as appropriate.
>>   *
>> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>   * 0 for success, error for failure.
>>   */
>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> -				struct amdgpu_vm *vm,
>> -				uint64_t saddr, uint64_t size)
>> +				struct amdgpu_vm *vm, uint64_t saddr,
>> +				uint64_t size, bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>>  	LIST_HEAD(removed);
>> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>  		    tmp->last = eaddr;
>>    		tmp->bo_va = NULL;
>> +		tmp->sync_unmap = sync_unmap;
>>  		list_add(&tmp->list, &vm->freed);
>>  		trace_amdgpu_vm_bo_unmap(NULL, tmp);
>>  	}
>> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>  		list_del(&mapping->list);
>>  		amdgpu_vm_it_remove(mapping, &vm->va);
>>  		mapping->bo_va = NULL;
>> +		mapping->sync_unmap = true;
>>  		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>  		list_add(&mapping->list, &vm->freed);
>>  	}
>> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
>>  		goto error_unlock;
>>  	}
>>  -	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
>> -				   addr, flags, value, 0, NULL, NULL, NULL);
>> +	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
>> +				   addr, addr, flags, value, 0, NULL, NULL,
>> +				   NULL);
>>  	if (r)
>>  		goto error_unlock;
>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 204ab13184ed..73b7b49fdb2e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  			   bool immediate, bool unlocked, bool flush_tlb,
>> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
>> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
>> +			   bool sync_unmap, struct dma_resv *resv,
>> +			   uint64_t start, uint64_t last, uint64_t flags,
>> +			   uint64_t offset, uint64_t vram_base,
>>  			   struct ttm_resource *res, dma_addr_t *pages_addr,
>>  			   struct dma_fence **fence);
>> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
>> -			struct amdgpu_bo_va *bo_va,
>> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  			bool clear);
>>  bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>>  void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>  		     uint64_t addr, uint64_t offset,
>>  		     uint64_t size, uint64_t flags);
>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>> -			     struct amdgpu_bo_va *bo_va,
>> -			     uint64_t addr, uint64_t offset,
>> -			     uint64_t size, uint64_t flags);
>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>> -		       struct amdgpu_bo_va *bo_va,
>> -		       uint64_t addr);
>> +			     struct amdgpu_bo_va *bo_va, uint64_t addr,
>> +			     uint64_t offset, uint64_t size, uint64_t flags,
>> +			     bool sync_unmap);
>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>> +		       uint64_t addr, bool sync_unmap);
>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> -				struct amdgpu_vm *vm,
>> -				uint64_t saddr, uint64_t size);
>> +				struct amdgpu_vm *vm, uint64_t saddr,
>> +				uint64_t size, bool sync_unmap);
>>  struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
>>  							 uint64_t addr);
>>  void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> index bb16b795d1bc..6eb4a0a4bc84 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>    	pr_debug("[0x%llx 0x%llx]\n", start, last);
>>  -	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
>> -				      last, init_pte_value, 0, 0, NULL, NULL,
>> -				      fence);
>> +	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
>> +				      start, last, init_pte_value, 0, 0, NULL,
>> +				      NULL, fence);
>>  }
>>    static int
>> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
>>  		 * different memory partition based on fpfn/lpfn, we should use
>>  		 * same vm_manager.vram_base_offset regardless memory partition.
>>  		 */
>> -		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
>> -					   last_start, prange->start + i,
>> -					   pte_flags,
>> -					   (last_start - prange->start) << PAGE_SHIFT,
>> -					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>> -					   NULL, dma_addr, &vm->last_update);
>> +		r = amdgpu_vm_update_range(
>> +			adev, vm, false, false, flush_tlb, true, NULL,
>> +			last_start, prange->start + i, pte_flags,
>> +			(last_start - prange->start) << PAGE_SHIFT,
>> +			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>> +			NULL, dma_addr, &vm->last_update);
>>    		for (j = last_start - prange->start; j <= i; j++)
>>  			dma_addr[j] |= last_domain;
>> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
>> index f477eda6a2b8..3cdcc299956e 100644
>> --- a/include/uapi/drm/amdgpu_drm.h
>> +++ b/include/uapi/drm/amdgpu_drm.h
>> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>>  #define AMDGPU_VM_MTYPE_RW		(5 << 5)
>>  /* don't allocate MALL */
>>  #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
>> +/* don't sync on unmap */
>> +#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
>>    struct drm_amdgpu_gem_va {
>>  	/** GEM object handle */


[-- Attachment #2: Type: text/html, Size: 70436 bytes --]

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Tatsuyuki Ishi]

[-- Attachment #1: Type: text/plain, Size: 20157 bytes --]

> On Nov 6, 2023, at 22:44, Christian König <christian.koenig@amd.com> wrote:
> 
>  Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
>> In Vulkan, it is the application's responsibility to perform adequate
>> synchronization before a sparse unmap, replace or BO destroy operation.
>> Until now, the kernel applied the same rule as implicitly-synchronized
>> APIs like OpenGL, which with per-VM BOs made page table updates stall the
>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>> drivers to opt-out of this behavior, while still ensuring adequate implicit
>> sync happens for kernel-initiated updates (e.g. BO moves).
>> 
>> We record whether to use implicit sync or not for each freed mapping. To
>> avoid increasing the mapping struct's size, this is union-ized with the
>> interval tree field which is unused after the unmap.
>> 
>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>> context global setting, is that the current libdrm implementation shares
>> the DRM handle even between different kind of drivers (radeonsi vs radv).
> 
> It would be nice if we could make this more future prove by not using a flag, but rather a drm_syncobj.

There is asynchronous VM_BIND and synchronous VM_BIND. Using syncobjs address asynchronous binds, but what this patch set solves is to add an explicitly synced synchronous bind.

Even within Vulkan, there are use cases for synchronous binds. This is when a non-sparse BO is destroyed (or created but that’s not synchronized). Such operations should still be explicit sync, unlike OpenGL where it syncs to previous submissions. So adding asynchronous bind doesn’t supersede this need.

I’ve also thought whether we can just make the unmap asynchronous, since the spec requires that destroyed stuff are not accessed in any way, but I think it will complicate behavior when the destruction of BO immediately follows.

We should implement asynchronous bind someday to make vkQueueBindSparse work (even) better, but that will likely involve a larger scope including the scheduler. Getting synchronous but explicitly synced binds should be simpler and a good incremental step.

Tatsuyuki.

> You can extend the drm_amdgpu_gem_va structure by adding a drm_syncobj handle and timeline point at the end.
> 
> If the syncobj/timeline point results in a fence we give that as input dependency the operation has to wait for.
> 
> And output fence can come later on as well, but that one is much more harder to handle.
> 
> Regards,
> Christian.
> 
>> 
>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>> ---
>>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>>  drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>>  include/uapi/drm/amdgpu_drm.h                 |  2 +
>>  9 files changed, 71 insertions(+), 50 deletions(-)
>> 
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> index 7d6daf8d2bfa..10e129bff977 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem *mem,
>>  	struct amdgpu_device *adev = entry->adev;
>>  	struct amdgpu_vm *vm = bo_va->base.vm;
>>  -	amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
>> +	amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>>    	amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> index 720011019741..612279e65bff 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  		}
>>  	}
>>  -	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
>> +	r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>>  	if (r) {
>>  		DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>>  		goto error;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> index a1b15d0d6c48..cca68b89754e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  	const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>>  		AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>>  		AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
>> -		AMDGPU_VM_PAGE_NOALLOC;
>> +		AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>>  	const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
>> -		AMDGPU_VM_PAGE_PRT;
>> +		AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>>    	struct drm_amdgpu_gem_va *args = data;
>>  	struct drm_gem_object *gobj;
>> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  	struct drm_exec exec;
>>  	uint64_t va_flags;
>>  	uint64_t vm_size;
>> +	bool sync_unmap;
>>  	int r = 0;
>>    	if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
>> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  		return -EINVAL;
>>  	}
>>  +	sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
>> +
>>  	switch (args->operation) {
>>  	case AMDGPU_VA_OP_MAP:
>>  	case AMDGPU_VA_OP_UNMAP:
>> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data,
>>  				     va_flags);
>>  		break;
>>  	case AMDGPU_VA_OP_UNMAP:
>> -		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
>> +		r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
>> +				       sync_unmap);
>>  		break;
>>    	case AMDGPU_VA_OP_CLEAR:
>>  		r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>>  						args->va_address,
>> -						args->map_size);
>> +						args->map_size, sync_unmap);
>>  		break;
>>  	case AMDGPU_VA_OP_REPLACE:
>>  		va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>>  		r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>>  					     args->offset_in_bo, args->map_size,
>> -					     va_flags);
>> +					     va_flags, sync_unmap);
>>  		break;
>>  	default:
>>  		break;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> index f3ee83cdf97e..28be03f1bbcf 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>>  	struct rb_node			rb;
>>  	uint64_t			start;
>>  	uint64_t			last;
>> -	uint64_t			__subtree_last;
>> +	union {
>> +		/* BOs in interval tree only */
>> +		uint64_t		__subtree_last;
>> +		/* Freed BOs only */
>> +		bool			sync_unmap;
>> +	};
>>  	uint64_t			offset;
>>  	uint64_t			flags;
>>  };
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> index 2fd1bfb35916..e71443c8c59b 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>  			     __field(long, last)
>>  			     __field(u64, offset)
>>  			     __field(u64, flags)
>> +			     __field(bool, sync_unmap)
>>  			     ),
>>    	    TP_fast_assign(
>> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>  			   __entry->last = mapping->last;
>>  			   __entry->offset = mapping->offset;
>>  			   __entry->flags = mapping->flags;
>> +			   __entry->sync_unmap = mapping->sync_unmap;
>>  			   ),
>> -	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
>> +	    TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx, sync_unmap=%d",
>>  		      __entry->bo, __entry->start, __entry->last,
>> -		      __entry->offset, __entry->flags)
>> +		      __entry->offset, __entry->flags, __entry->sync_unmap)
>>  );
>>    DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> index 7b9762f1cddd..a74472e16952 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>>   * @immediate: immediate submission in a page fault
>>   * @unlocked: unlocked invalidation during MM callback
>>   * @flush_tlb: trigger tlb invalidation after update completed
>> + * @sync_unmap: wait for BO users before unmapping
>>   * @resv: fences we need to sync to
>>   * @start: start of mapped range
>>   * @last: last mapped entry
>> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct dma_fence *fence,
>>   */
>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  			   bool immediate, bool unlocked, bool flush_tlb,
>> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
>> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
>> +			   bool sync_unmap, struct dma_resv *resv,
>> +			   uint64_t start, uint64_t last, uint64_t flags,
>> +			   uint64_t offset, uint64_t vram_base,
>>  			   struct ttm_resource *res, dma_addr_t *pages_addr,
>>  			   struct dma_fence **fence)
>>  {
>> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  	/* Implicitly sync to command submissions in the same VM before
>>  	 * unmapping. Sync to moving fences before mapping.
>>  	 */
>> -	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
>> +	if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>>  		sync_mode = AMDGPU_SYNC_EQ_OWNER;
>>  	else
>>  		sync_mode = AMDGPU_SYNC_EXPLICIT;
>> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  		trace_amdgpu_vm_bo_update(mapping);
>>    		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
>> -					   resv, mapping->start, mapping->last,
>> -					   update_flags, mapping->offset,
>> -					   vram_base, mem, pages_addr,
>> -					   last_update);
>> +					   true, resv, mapping->start,
>> +					   mapping->last, update_flags,
>> +					   mapping->offset, vram_base, mem,
>> +					   pages_addr, last_update);
>>  		if (r)
>>  			return r;
>>  	}
>> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
>>  		    mapping->start < AMDGPU_GMC_HOLE_START)
>>  			init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>>  -		r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
>> +		r = amdgpu_vm_update_range(adev, vm, false, false, true,
>> +					   mapping->sync_unmap, resv,
>>  					   mapping->start, mapping->last,
>>  					   init_pte_value, 0, 0, NULL, NULL,
>>  					   &f);
>> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>   * @offset: requested offset in the BO
>>   * @size: BO size in bytes
>>   * @flags: attributes of pages (read/write/valid/etc.)
>> + * @sync_unmap: wait for BO users before replacing existing mapping
>>   *
>>   * Add a mapping of the BO at the specefied addr into the VM. Replace existing
>>   * mappings as we do so.
>> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>   * Object has to be reserved and unreserved outside!
>>   */
>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>> -			     struct amdgpu_bo_va *bo_va,
>> -			     uint64_t saddr, uint64_t offset,
>> -			     uint64_t size, uint64_t flags)
>> +			     struct amdgpu_bo_va *bo_va, uint64_t saddr,
>> +			     uint64_t offset, uint64_t size, uint64_t flags,
>> +			     bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *mapping;
>>  	struct amdgpu_bo *bo = bo_va->base.bo;
>> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>  	if (!mapping)
>>  		return -ENOMEM;
>>  -	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
>> +	r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, sync_unmap);
>>  	if (r) {
>>  		kfree(mapping);
>>  		return r;
>> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>   * @adev: amdgpu_device pointer
>>   * @bo_va: bo_va to remove the address from
>>   * @saddr: where to the BO is mapped
>> + * @sync_unmap: wait for BO users before unmapping
>>   *
>>   * Remove a mapping of the BO at the specefied addr from the VM.
>>   *
>> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>   *
>>   * Object has to be reserved and unreserved outside!
>>   */
>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>> -		       struct amdgpu_bo_va *bo_va,
>> -		       uint64_t saddr)
>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>> +		       uint64_t saddr, bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *mapping;
>>  	struct amdgpu_vm *vm = bo_va->base.vm;
>> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>  	list_del(&mapping->list);
>>  	amdgpu_vm_it_remove(mapping, &vm->va);
>>  	mapping->bo_va = NULL;
>> +	mapping->sync_unmap = sync_unmap;
>>  	trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>    	if (valid)
>> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>   * @vm: VM structure to use
>>   * @saddr: start of the range
>>   * @size: size of the range
>> + * @sync_unmap: wait for BO users before unmapping
>>   *
>>   * Remove all mappings in a range, split them as appropriate.
>>   *
>> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>   * 0 for success, error for failure.
>>   */
>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> -				struct amdgpu_vm *vm,
>> -				uint64_t saddr, uint64_t size)
>> +				struct amdgpu_vm *vm, uint64_t saddr,
>> +				uint64_t size, bool sync_unmap)
>>  {
>>  	struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>>  	LIST_HEAD(removed);
>> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>  		    tmp->last = eaddr;
>>    		tmp->bo_va = NULL;
>> +		tmp->sync_unmap = sync_unmap;
>>  		list_add(&tmp->list, &vm->freed);
>>  		trace_amdgpu_vm_bo_unmap(NULL, tmp);
>>  	}
>> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>  		list_del(&mapping->list);
>>  		amdgpu_vm_it_remove(mapping, &vm->va);
>>  		mapping->bo_va = NULL;
>> +		mapping->sync_unmap = true;
>>  		trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>  		list_add(&mapping->list, &vm->freed);
>>  	}
>> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
>>  		goto error_unlock;
>>  	}
>>  -	r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
>> -				   addr, flags, value, 0, NULL, NULL, NULL);
>> +	r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
>> +				   addr, addr, flags, value, 0, NULL, NULL,
>> +				   NULL);
>>  	if (r)
>>  		goto error_unlock;
>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 204ab13184ed..73b7b49fdb2e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base,
>>  			    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>  			   bool immediate, bool unlocked, bool flush_tlb,
>> -			   struct dma_resv *resv, uint64_t start, uint64_t last,
>> -			   uint64_t flags, uint64_t offset, uint64_t vram_base,
>> +			   bool sync_unmap, struct dma_resv *resv,
>> +			   uint64_t start, uint64_t last, uint64_t flags,
>> +			   uint64_t offset, uint64_t vram_base,
>>  			   struct ttm_resource *res, dma_addr_t *pages_addr,
>>  			   struct dma_fence **fence);
>> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
>> -			struct amdgpu_bo_va *bo_va,
>> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>>  			bool clear);
>>  bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>>  void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>  		     uint64_t addr, uint64_t offset,
>>  		     uint64_t size, uint64_t flags);
>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>> -			     struct amdgpu_bo_va *bo_va,
>> -			     uint64_t addr, uint64_t offset,
>> -			     uint64_t size, uint64_t flags);
>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>> -		       struct amdgpu_bo_va *bo_va,
>> -		       uint64_t addr);
>> +			     struct amdgpu_bo_va *bo_va, uint64_t addr,
>> +			     uint64_t offset, uint64_t size, uint64_t flags,
>> +			     bool sync_unmap);
>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
>> +		       uint64_t addr, bool sync_unmap);
>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>> -				struct amdgpu_vm *vm,
>> -				uint64_t saddr, uint64_t size);
>> +				struct amdgpu_vm *vm, uint64_t saddr,
>> +				uint64_t size, bool sync_unmap);
>>  struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct amdgpu_vm *vm,
>>  							 uint64_t addr);
>>  void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct ww_acquire_ctx *ticket);
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> index bb16b795d1bc..6eb4a0a4bc84 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
>>    	pr_debug("[0x%llx 0x%llx]\n", start, last);
>>  -	return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, start,
>> -				      last, init_pte_value, 0, 0, NULL, NULL,
>> -				      fence);
>> +	return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
>> +				      start, last, init_pte_value, 0, 0, NULL,
>> +				      NULL, fence);
>>  }
>>    static int
>> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
>>  		 * different memory partition based on fpfn/lpfn, we should use
>>  		 * same vm_manager.vram_base_offset regardless memory partition.
>>  		 */
>> -		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
>> -					   last_start, prange->start + i,
>> -					   pte_flags,
>> -					   (last_start - prange->start) << PAGE_SHIFT,
>> -					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>> -					   NULL, dma_addr, &vm->last_update);
>> +		r = amdgpu_vm_update_range(
>> +			adev, vm, false, false, flush_tlb, true, NULL,
>> +			last_start, prange->start + i, pte_flags,
>> +			(last_start - prange->start) << PAGE_SHIFT,
>> +			bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>> +			NULL, dma_addr, &vm->last_update);
>>    		for (j = last_start - prange->start; j <= i; j++)
>>  			dma_addr[j] |= last_domain;
>> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
>> index f477eda6a2b8..3cdcc299956e 100644
>> --- a/include/uapi/drm/amdgpu_drm.h
>> +++ b/include/uapi/drm/amdgpu_drm.h
>> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>>  #define AMDGPU_VM_MTYPE_RW		(5 << 5)
>>  /* don't allocate MALL */
>>  #define AMDGPU_VM_PAGE_NOALLOC		(1 << 9)
>> +/* don't sync on unmap */
>> +#define AMDGPU_VM_EXPLICIT_SYNC		(1 << 10)
>>    struct drm_amdgpu_gem_va {
>>  	/** GEM object handle */


[-- Attachment #2: Type: text/html, Size: 70436 bytes --]

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 21029 bytes --]

Am 06.11.23 um 16:47 schrieb Tatsuyuki Ishi:
>> On Nov 6, 2023, at 22:44, Christian König <christian.koenig@amd.com> 
>> wrote:
>>
>> Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
>>> In Vulkan, it is the application's responsibility to perform adequate
>>> synchronization before a sparse unmap, replace or BO destroy operation.
>>> Until now, the kernel applied the same rule as implicitly-synchronized
>>> APIs like OpenGL, which with per-VM BOs made page table updates 
>>> stall the
>>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>>> drivers to opt-out of this behavior, while still ensuring adequate 
>>> implicit
>>> sync happens for kernel-initiated updates (e.g. BO moves).
>>>
>>> We record whether to use implicit sync or not for each freed mapping. To
>>> avoid increasing the mapping struct's size, this is union-ized with the
>>> interval tree field which is unused after the unmap.
>>>
>>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>>> context global setting, is that the current libdrm implementation shares
>>> the DRM handle even between different kind of drivers (radeonsi vs 
>>> radv).
>>
>> It would be nice if we could make this more future prove by not using 
>> a flag, but rather a drm_syncobj.
>
> There is asynchronous VM_BIND and synchronous VM_BIND. Using syncobjs 
> address asynchronous binds, but what this patch set solves is to add 
> an explicitly synced synchronous bind.

All VM updates are asynchronous in the sense that they are queues up and 
don't execute immediately.

If you don't add input/output fences and don't sync implicitly with 
command submission any more you actually have no idea in userspace when 
they execute.

That doesn't sound like a good idea to me.

>
> Even within Vulkan, there are use cases for synchronous binds. This is 
> when a non-sparse BO is destroyed (or created but that’s not 
> synchronized). Such operations should still be explicit sync, unlike 
> OpenGL where it syncs to previous submissions. So adding asynchronous 
> bind doesn’t supersede this need.
>
> I’ve also thought whether we can just make the unmap asynchronous, 
> since the spec requires that destroyed stuff are not accessed in any 
> way, but I think it will complicate behavior when the destruction of 
> BO immediately follows.
>
> We should implement asynchronous bind someday to make 
> vkQueueBindSparse work (even) better, but that will likely involve a 
> larger scope including the scheduler. Getting synchronous but 
> explicitly synced binds should be simpler and a good incremental step.

That's the whole point, I don't think that the flag is simpler/cleaner 
than a fence.

We still need to take the implicit sync which can come from kernel 
operations into account, but at the same time disable the implicit sync 
which comes from user space submissions.

As far as I can see the easiest way to do this and which both doesn't 
break anything currently and still leaves room to extend the interface 
is to add an input dependency fence.

If you then use a signaled syncpoint as input you get exactly the 
semantic you desire while we are still able to add an output fence later on.

Regards,
Christian.

>
> Tatsuyuki.
>
>> You can extend the drm_amdgpu_gem_va structure by adding a 
>> drm_syncobj handle and timeline point at the end.
>>
>> If the syncobj/timeline point results in a fence we give that as 
>> input dependency the operation has to wait for.
>>
>> And output fence can come later on as well, but that one is much more 
>> harder to handle.
>>
>> Regards,
>> Christian.
>>
>>>
>>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>> ---
>>>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>>>  drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>>>  include/uapi/drm/amdgpu_drm.h                 |  2 +
>>>  9 files changed, 71 insertions(+), 50 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> index 7d6daf8d2bfa..10e129bff977 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem 
>>> *mem,
>>> struct amdgpu_device *adev = entry->adev;
>>> struct amdgpu_vm *vm = bo_va->base.vm;
>>>  -amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
>>> +amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>>> amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> index 720011019741..612279e65bff 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> }
>>> }
>>>  -r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
>>> +r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>>> if (r) {
>>> DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>>> goto error;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> index a1b15d0d6c48..cca68b89754e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>>> AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>>> AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
>>> -AMDGPU_VM_PAGE_NOALLOC;
>>> +AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>>> const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
>>> -AMDGPU_VM_PAGE_PRT;
>>> +AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>>> struct drm_amdgpu_gem_va *args = data;
>>> struct drm_gem_object *gobj;
>>> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> struct drm_exec exec;
>>> uint64_t va_flags;
>>> uint64_t vm_size;
>>> +bool sync_unmap;
>>> int r = 0;
>>> if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
>>> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> return -EINVAL;
>>> }
>>>  +sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
>>> +
>>> switch (args->operation) {
>>> case AMDGPU_VA_OP_MAP:
>>> case AMDGPU_VA_OP_UNMAP:
>>> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device 
>>> *dev, void *data,
>>>     va_flags);
>>> break;
>>> case AMDGPU_VA_OP_UNMAP:
>>> -r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
>>> +r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
>>> +      sync_unmap);
>>> break;
>>> case AMDGPU_VA_OP_CLEAR:
>>> r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>>> args->va_address,
>>> -args->map_size);
>>> +args->map_size, sync_unmap);
>>> break;
>>> case AMDGPU_VA_OP_REPLACE:
>>> va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>>> r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>>>     args->offset_in_bo, args->map_size,
>>> -    va_flags);
>>> +    va_flags, sync_unmap);
>>> break;
>>> default:
>>> break;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> index f3ee83cdf97e..28be03f1bbcf 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>>> struct rb_noderb;
>>> uint64_tstart;
>>> uint64_tlast;
>>> -uint64_t__subtree_last;
>>> +union {
>>> +/* BOs in interval tree only */
>>> +uint64_t__subtree_last;
>>> +/* Freed BOs only */
>>> +boolsync_unmap;
>>> +};
>>> uint64_toffset;
>>> uint64_tflags;
>>>  };
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> index 2fd1bfb35916..e71443c8c59b 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>>     __field(long, last)
>>>     __field(u64, offset)
>>>     __field(u64, flags)
>>> +    __field(bool, sync_unmap)
>>>     ),
>>>    TP_fast_assign(
>>> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>>   __entry->last = mapping->last;
>>>   __entry->offset = mapping->offset;
>>>   __entry->flags = mapping->flags;
>>> +  __entry->sync_unmap = mapping->sync_unmap;
>>>   ),
>>> -   TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
>>> +   TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, 
>>> flags=%llx, sync_unmap=%d",
>>>      __entry->bo, __entry->start, __entry->last,
>>> -     __entry->offset, __entry->flags)
>>> +     __entry->offset, __entry->flags, __entry->sync_unmap)
>>>  );
>>>    DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index 7b9762f1cddd..a74472e16952 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct 
>>> dma_fence *fence,
>>>   * @immediate: immediate submission in a page fault
>>>   * @unlocked: unlocked invalidation during MM callback
>>>   * @flush_tlb: trigger tlb invalidation after update completed
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   * @resv: fences we need to sync to
>>>   * @start: start of mapped range
>>>   * @last: last mapped entry
>>> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct 
>>> dma_fence *fence,
>>>   */
>>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct 
>>> amdgpu_vm *vm,
>>>   bool immediate, bool unlocked, bool flush_tlb,
>>> -  struct dma_resv *resv, uint64_t start, uint64_t last,
>>> -  uint64_t flags, uint64_t offset, uint64_t vram_base,
>>> +  bool sync_unmap, struct dma_resv *resv,
>>> +  uint64_t start, uint64_t last, uint64_t flags,
>>> +  uint64_t offset, uint64_t vram_base,
>>>   struct ttm_resource *res, dma_addr_t *pages_addr,
>>>   struct dma_fence **fence)
>>>  {
>>> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> /* Implicitly sync to command submissions in the same VM before
>>> * unmapping. Sync to moving fences before mapping.
>>> */
>>> -if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
>>> +if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>>> sync_mode = AMDGPU_SYNC_EQ_OWNER;
>>> else
>>> sync_mode = AMDGPU_SYNC_EXPLICIT;
>>> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device 
>>> *adev, struct amdgpu_bo_va *bo_va,
>>> trace_amdgpu_vm_bo_update(mapping);
>>> r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
>>> -  resv, mapping->start, mapping->last,
>>> -  update_flags, mapping->offset,
>>> -  vram_base, mem, pages_addr,
>>> -  last_update);
>>> +  true, resv, mapping->start,
>>> +  mapping->last, update_flags,
>>> +  mapping->offset, vram_base, mem,
>>> +  pages_addr, last_update);
>>> if (r)
>>> return r;
>>> }
>>> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device 
>>> *adev,
>>>    mapping->start < AMDGPU_GMC_HOLE_START)
>>> init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>>>  -r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
>>> +r = amdgpu_vm_update_range(adev, vm, false, false, true,
>>> +  mapping->sync_unmap, resv,
>>>   mapping->start, mapping->last,
>>>   init_pte_value, 0, 0, NULL, NULL,
>>>   &f);
>>> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>   * @offset: requested offset in the BO
>>>   * @size: BO size in bytes
>>>   * @flags: attributes of pages (read/write/valid/etc.)
>>> + * @sync_unmap: wait for BO users before replacing existing mapping
>>>   *
>>>   * Add a mapping of the BO at the specefied addr into the VM. 
>>> Replace existing
>>>   * mappings as we do so.
>>> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>   * Object has to be reserved and unreserved outside!
>>>   */
>>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>> -    struct amdgpu_bo_va *bo_va,
>>> -    uint64_t saddr, uint64_t offset,
>>> -    uint64_t size, uint64_t flags)
>>> +    struct amdgpu_bo_va *bo_va, uint64_t saddr,
>>> +    uint64_t offset, uint64_t size, uint64_t flags,
>>> +    bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *mapping;
>>> struct amdgpu_bo *bo = bo_va->base.bo;
>>> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>> if (!mapping)
>>> return -ENOMEM;
>>>  -r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
>>> +r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, 
>>> sync_unmap);
>>> if (r) {
>>> kfree(mapping);
>>> return r;
>>> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>>   * @adev: amdgpu_device pointer
>>>   * @bo_va: bo_va to remove the address from
>>>   * @saddr: where to the BO is mapped
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   *
>>>   * Remove a mapping of the BO at the specefied addr from the VM.
>>>   *
>>> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>>   *
>>>   * Object has to be reserved and unreserved outside!
>>>   */
>>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> -      struct amdgpu_bo_va *bo_va,
>>> -      uint64_t saddr)
>>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> +      uint64_t saddr, bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *mapping;
>>> struct amdgpu_vm *vm = bo_va->base.vm;
>>> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> list_del(&mapping->list);
>>> amdgpu_vm_it_remove(mapping, &vm->va);
>>> mapping->bo_va = NULL;
>>> +mapping->sync_unmap = sync_unmap;
>>> trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>> if (valid)
>>> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>>   * @vm: VM structure to use
>>>   * @saddr: start of the range
>>>   * @size: size of the range
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   *
>>>   * Remove all mappings in a range, split them as appropriate.
>>>   *
>>> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>>   * 0 for success, error for failure.
>>>   */
>>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> -struct amdgpu_vm *vm,
>>> -uint64_t saddr, uint64_t size)
>>> +struct amdgpu_vm *vm, uint64_t saddr,
>>> +uint64_t size, bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>>> LIST_HEAD(removed);
>>> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct 
>>> amdgpu_device *adev,
>>>    tmp->last = eaddr;
>>> tmp->bo_va = NULL;
>>> +tmp->sync_unmap = sync_unmap;
>>> list_add(&tmp->list, &vm->freed);
>>> trace_amdgpu_vm_bo_unmap(NULL, tmp);
>>> }
>>> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>> list_del(&mapping->list);
>>> amdgpu_vm_it_remove(mapping, &vm->va);
>>> mapping->bo_va = NULL;
>>> +mapping->sync_unmap = true;
>>> trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>> list_add(&mapping->list, &vm->freed);
>>> }
>>> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct 
>>> amdgpu_device *adev, u32 pasid,
>>> goto error_unlock;
>>> }
>>>  -r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
>>> -  addr, flags, value, 0, NULL, NULL, NULL);
>>> +r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
>>> +  addr, addr, flags, value, 0, NULL, NULL,
>>> +  NULL);
>>> if (r)
>>> goto error_unlock;
>>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index 204ab13184ed..73b7b49fdb2e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct 
>>> amdgpu_vm_bo_base *base,
>>>    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct 
>>> amdgpu_vm *vm,
>>>   bool immediate, bool unlocked, bool flush_tlb,
>>> -  struct dma_resv *resv, uint64_t start, uint64_t last,
>>> -  uint64_t flags, uint64_t offset, uint64_t vram_base,
>>> +  bool sync_unmap, struct dma_resv *resv,
>>> +  uint64_t start, uint64_t last, uint64_t flags,
>>> +  uint64_t offset, uint64_t vram_base,
>>>   struct ttm_resource *res, dma_addr_t *pages_addr,
>>>   struct dma_fence **fence);
>>> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
>>> -struct amdgpu_bo_va *bo_va,
>>> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> bool clear);
>>>  bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>>>  void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>     uint64_t addr, uint64_t offset,
>>>     uint64_t size, uint64_t flags);
>>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>> -    struct amdgpu_bo_va *bo_va,
>>> -    uint64_t addr, uint64_t offset,
>>> -    uint64_t size, uint64_t flags);
>>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> -      struct amdgpu_bo_va *bo_va,
>>> -      uint64_t addr);
>>> +    struct amdgpu_bo_va *bo_va, uint64_t addr,
>>> +    uint64_t offset, uint64_t size, uint64_t flags,
>>> +    bool sync_unmap);
>>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> +      uint64_t addr, bool sync_unmap);
>>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> -struct amdgpu_vm *vm,
>>> -uint64_t saddr, uint64_t size);
>>> +struct amdgpu_vm *vm, uint64_t saddr,
>>> +uint64_t size, bool sync_unmap);
>>>  struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct 
>>> amdgpu_vm *vm,
>>> uint64_t addr);
>>>  void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct 
>>> ww_acquire_ctx *ticket);
>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c 
>>> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> index bb16b795d1bc..6eb4a0a4bc84 100644
>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> pr_debug("[0x%llx 0x%llx]\n", start, last);
>>>  -return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, 
>>> start,
>>> -     last, init_pte_value, 0, 0, NULL, NULL,
>>> -     fence);
>>> +return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
>>> +     start, last, init_pte_value, 0, 0, NULL,
>>> +     NULL, fence);
>>>  }
>>>    static int
>>> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct 
>>> kfd_process_device *pdd, struct svm_range *prange,
>>> * different memory partition based on fpfn/lpfn, we should use
>>> * same vm_manager.vram_base_offset regardless memory partition.
>>> */
>>> -r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
>>> -  last_start, prange->start + i,
>>> -  pte_flags,
>>> -  (last_start - prange->start) << PAGE_SHIFT,
>>> -  bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>>> -  NULL, dma_addr, &vm->last_update);
>>> +r = amdgpu_vm_update_range(
>>> +adev, vm, false, false, flush_tlb, true, NULL,
>>> +last_start, prange->start + i, pte_flags,
>>> +(last_start - prange->start) << PAGE_SHIFT,
>>> +bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>>> +NULL, dma_addr, &vm->last_update);
>>> for (j = last_start - prange->start; j <= i; j++)
>>> dma_addr[j] |= last_domain;
>>> diff --git a/include/uapi/drm/amdgpu_drm.h 
>>> b/include/uapi/drm/amdgpu_drm.h
>>> index f477eda6a2b8..3cdcc299956e 100644
>>> --- a/include/uapi/drm/amdgpu_drm.h
>>> +++ b/include/uapi/drm/amdgpu_drm.h
>>> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>>>  #define AMDGPU_VM_MTYPE_RW(5 << 5)
>>>  /* don't allocate MALL */
>>>  #define AMDGPU_VM_PAGE_NOALLOC(1 << 9)
>>> +/* don't sync on unmap */
>>> +#define AMDGPU_VM_EXPLICIT_SYNC(1 << 10)
>>>    struct drm_amdgpu_gem_va {
>>> /** GEM object handle */
>

[-- Attachment #2: Type: text/html, Size: 85941 bytes --]

Re: [PATCH v2 2/3] drm/amdgpu: Add flag to disable implicit sync for GEM operations.

[Christian König]

[-- Attachment #1: Type: text/plain, Size: 21029 bytes --]

Am 06.11.23 um 16:47 schrieb Tatsuyuki Ishi:
>> On Nov 6, 2023, at 22:44, Christian König <christian.koenig@amd.com> 
>> wrote:
>>
>> Am 02.11.23 um 15:04 schrieb Tatsuyuki Ishi:
>>> In Vulkan, it is the application's responsibility to perform adequate
>>> synchronization before a sparse unmap, replace or BO destroy operation.
>>> Until now, the kernel applied the same rule as implicitly-synchronized
>>> APIs like OpenGL, which with per-VM BOs made page table updates 
>>> stall the
>>> queue completely. The newly added AMDGPU_VM_EXPLICIT_SYNC flag allows
>>> drivers to opt-out of this behavior, while still ensuring adequate 
>>> implicit
>>> sync happens for kernel-initiated updates (e.g. BO moves).
>>>
>>> We record whether to use implicit sync or not for each freed mapping. To
>>> avoid increasing the mapping struct's size, this is union-ized with the
>>> interval tree field which is unused after the unmap.
>>>
>>> The reason this is done with a GEM ioctl flag, instead of being a VM /
>>> context global setting, is that the current libdrm implementation shares
>>> the DRM handle even between different kind of drivers (radeonsi vs 
>>> radv).
>>
>> It would be nice if we could make this more future prove by not using 
>> a flag, but rather a drm_syncobj.
>
> There is asynchronous VM_BIND and synchronous VM_BIND. Using syncobjs 
> address asynchronous binds, but what this patch set solves is to add 
> an explicitly synced synchronous bind.

All VM updates are asynchronous in the sense that they are queues up and 
don't execute immediately.

If you don't add input/output fences and don't sync implicitly with 
command submission any more you actually have no idea in userspace when 
they execute.

That doesn't sound like a good idea to me.

>
> Even within Vulkan, there are use cases for synchronous binds. This is 
> when a non-sparse BO is destroyed (or created but that’s not 
> synchronized). Such operations should still be explicit sync, unlike 
> OpenGL where it syncs to previous submissions. So adding asynchronous 
> bind doesn’t supersede this need.
>
> I’ve also thought whether we can just make the unmap asynchronous, 
> since the spec requires that destroyed stuff are not accessed in any 
> way, but I think it will complicate behavior when the destruction of 
> BO immediately follows.
>
> We should implement asynchronous bind someday to make 
> vkQueueBindSparse work (even) better, but that will likely involve a 
> larger scope including the scheduler. Getting synchronous but 
> explicitly synced binds should be simpler and a good incremental step.

That's the whole point, I don't think that the flag is simpler/cleaner 
than a fence.

We still need to take the implicit sync which can come from kernel 
operations into account, but at the same time disable the implicit sync 
which comes from user space submissions.

As far as I can see the easiest way to do this and which both doesn't 
break anything currently and still leaves room to extend the interface 
is to add an input dependency fence.

If you then use a signaled syncpoint as input you get exactly the 
semantic you desire while we are still able to add an output fence later on.

Regards,
Christian.

>
> Tatsuyuki.
>
>> You can extend the drm_amdgpu_gem_va structure by adding a 
>> drm_syncobj handle and timeline point at the end.
>>
>> If the syncobj/timeline point results in a fence we give that as 
>> input dependency the operation has to wait for.
>>
>> And output fence can come later on as well, but that one is much more 
>> harder to handle.
>>
>> Regards,
>> Christian.
>>
>>>
>>> Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
>>> ---
>>>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c       |  2 +-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 14 ++++--
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  7 ++-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  6 ++-
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 47 +++++++++++--------
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        | 23 +++++----
>>>  drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 18 +++----
>>>  include/uapi/drm/amdgpu_drm.h                 |  2 +
>>>  9 files changed, 71 insertions(+), 50 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> index 7d6daf8d2bfa..10e129bff977 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
>>> @@ -1196,7 +1196,7 @@ static void unmap_bo_from_gpuvm(struct kgd_mem 
>>> *mem,
>>> struct amdgpu_device *adev = entry->adev;
>>> struct amdgpu_vm *vm = bo_va->base.vm;
>>>  -amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
>>> +amdgpu_vm_bo_unmap(adev, bo_va, entry->va, true);
>>> amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
>>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> index 720011019741..612279e65bff 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
>>> @@ -122,7 +122,7 @@ int amdgpu_unmap_static_csa(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> }
>>> }
>>>  -r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr);
>>> +r = amdgpu_vm_bo_unmap(adev, bo_va, csa_addr, true);
>>> if (r) {
>>> DRM_ERROR("failed to do bo_unmap on static CSA, err=%d\n", r);
>>> goto error;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> index a1b15d0d6c48..cca68b89754e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
>>> @@ -667,9 +667,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> const uint32_t valid_flags = AMDGPU_VM_DELAY_UPDATE |
>>> AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE |
>>> AMDGPU_VM_PAGE_EXECUTABLE | AMDGPU_VM_MTYPE_MASK |
>>> -AMDGPU_VM_PAGE_NOALLOC;
>>> +AMDGPU_VM_PAGE_NOALLOC | AMDGPU_VM_EXPLICIT_SYNC;
>>> const uint32_t prt_flags = AMDGPU_VM_DELAY_UPDATE |
>>> -AMDGPU_VM_PAGE_PRT;
>>> +AMDGPU_VM_PAGE_PRT | AMDGPU_VM_EXPLICIT_SYNC;
>>> struct drm_amdgpu_gem_va *args = data;
>>> struct drm_gem_object *gobj;
>>> @@ -680,6 +680,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> struct drm_exec exec;
>>> uint64_t va_flags;
>>> uint64_t vm_size;
>>> +bool sync_unmap;
>>> int r = 0;
>>> if (args->va_address < AMDGPU_VA_RESERVED_SIZE) {
>>> @@ -715,6 +716,8 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, 
>>> void *data,
>>> return -EINVAL;
>>> }
>>>  +sync_unmap = !(args->flags & AMDGPU_VM_EXPLICIT_SYNC);
>>> +
>>> switch (args->operation) {
>>> case AMDGPU_VA_OP_MAP:
>>> case AMDGPU_VA_OP_UNMAP:
>>> @@ -774,19 +777,20 @@ int amdgpu_gem_va_ioctl(struct drm_device 
>>> *dev, void *data,
>>>     va_flags);
>>> break;
>>> case AMDGPU_VA_OP_UNMAP:
>>> -r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address);
>>> +r = amdgpu_vm_bo_unmap(adev, bo_va, args->va_address,
>>> +      sync_unmap);
>>> break;
>>> case AMDGPU_VA_OP_CLEAR:
>>> r = amdgpu_vm_bo_clear_mappings(adev, &fpriv->vm,
>>> args->va_address,
>>> -args->map_size);
>>> +args->map_size, sync_unmap);
>>> break;
>>> case AMDGPU_VA_OP_REPLACE:
>>> va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
>>> r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
>>>     args->offset_in_bo, args->map_size,
>>> -    va_flags);
>>> +    va_flags, sync_unmap);
>>> break;
>>> default:
>>> break;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> index f3ee83cdf97e..28be03f1bbcf 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
>>> @@ -67,7 +67,12 @@ struct amdgpu_bo_va_mapping {
>>> struct rb_noderb;
>>> uint64_tstart;
>>> uint64_tlast;
>>> -uint64_t__subtree_last;
>>> +union {
>>> +/* BOs in interval tree only */
>>> +uint64_t__subtree_last;
>>> +/* Freed BOs only */
>>> +boolsync_unmap;
>>> +};
>>> uint64_toffset;
>>> uint64_tflags;
>>>  };
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> index 2fd1bfb35916..e71443c8c59b 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>> @@ -276,6 +276,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>>     __field(long, last)
>>>     __field(u64, offset)
>>>     __field(u64, flags)
>>> +    __field(bool, sync_unmap)
>>>     ),
>>>    TP_fast_assign(
>>> @@ -284,10 +285,11 @@ TRACE_EVENT(amdgpu_vm_bo_unmap,
>>>   __entry->last = mapping->last;
>>>   __entry->offset = mapping->offset;
>>>   __entry->flags = mapping->flags;
>>> +  __entry->sync_unmap = mapping->sync_unmap;
>>>   ),
>>> -   TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, flags=%llx",
>>> +   TP_printk("bo=%p, start=%lx, last=%lx, offset=%010llx, 
>>> flags=%llx, sync_unmap=%d",
>>>      __entry->bo, __entry->start, __entry->last,
>>> -     __entry->offset, __entry->flags)
>>> +     __entry->offset, __entry->flags, __entry->sync_unmap)
>>>  );
>>>    DECLARE_EVENT_CLASS(amdgpu_vm_mapping,
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index 7b9762f1cddd..a74472e16952 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -844,6 +844,7 @@ static void amdgpu_vm_tlb_seq_cb(struct 
>>> dma_fence *fence,
>>>   * @immediate: immediate submission in a page fault
>>>   * @unlocked: unlocked invalidation during MM callback
>>>   * @flush_tlb: trigger tlb invalidation after update completed
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   * @resv: fences we need to sync to
>>>   * @start: start of mapped range
>>>   * @last: last mapped entry
>>> @@ -861,8 +862,9 @@ static void amdgpu_vm_tlb_seq_cb(struct 
>>> dma_fence *fence,
>>>   */
>>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct 
>>> amdgpu_vm *vm,
>>>   bool immediate, bool unlocked, bool flush_tlb,
>>> -  struct dma_resv *resv, uint64_t start, uint64_t last,
>>> -  uint64_t flags, uint64_t offset, uint64_t vram_base,
>>> +  bool sync_unmap, struct dma_resv *resv,
>>> +  uint64_t start, uint64_t last, uint64_t flags,
>>> +  uint64_t offset, uint64_t vram_base,
>>>   struct ttm_resource *res, dma_addr_t *pages_addr,
>>>   struct dma_fence **fence)
>>>  {
>>> @@ -902,7 +904,7 @@ int amdgpu_vm_update_range(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> /* Implicitly sync to command submissions in the same VM before
>>> * unmapping. Sync to moving fences before mapping.
>>> */
>>> -if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)))
>>> +if (!(flags & (AMDGPU_PTE_VALID | AMDGPU_PTE_PRT)) && sync_unmap)
>>> sync_mode = AMDGPU_SYNC_EQ_OWNER;
>>> else
>>> sync_mode = AMDGPU_SYNC_EXPLICIT;
>>> @@ -1145,10 +1147,10 @@ int amdgpu_vm_bo_update(struct amdgpu_device 
>>> *adev, struct amdgpu_bo_va *bo_va,
>>> trace_amdgpu_vm_bo_update(mapping);
>>> r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb,
>>> -  resv, mapping->start, mapping->last,
>>> -  update_flags, mapping->offset,
>>> -  vram_base, mem, pages_addr,
>>> -  last_update);
>>> +  true, resv, mapping->start,
>>> +  mapping->last, update_flags,
>>> +  mapping->offset, vram_base, mem,
>>> +  pages_addr, last_update);
>>> if (r)
>>> return r;
>>> }
>>> @@ -1340,7 +1342,8 @@ int amdgpu_vm_clear_freed(struct amdgpu_device 
>>> *adev,
>>>    mapping->start < AMDGPU_GMC_HOLE_START)
>>> init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>>>  -r = amdgpu_vm_update_range(adev, vm, false, false, true, resv,
>>> +r = amdgpu_vm_update_range(adev, vm, false, false, true,
>>> +  mapping->sync_unmap, resv,
>>>   mapping->start, mapping->last,
>>>   init_pte_value, 0, 0, NULL, NULL,
>>>   &f);
>>> @@ -1572,6 +1575,7 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>   * @offset: requested offset in the BO
>>>   * @size: BO size in bytes
>>>   * @flags: attributes of pages (read/write/valid/etc.)
>>> + * @sync_unmap: wait for BO users before replacing existing mapping
>>>   *
>>>   * Add a mapping of the BO at the specefied addr into the VM. 
>>> Replace existing
>>>   * mappings as we do so.
>>> @@ -1582,9 +1586,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>   * Object has to be reserved and unreserved outside!
>>>   */
>>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>> -    struct amdgpu_bo_va *bo_va,
>>> -    uint64_t saddr, uint64_t offset,
>>> -    uint64_t size, uint64_t flags)
>>> +    struct amdgpu_bo_va *bo_va, uint64_t saddr,
>>> +    uint64_t offset, uint64_t size, uint64_t flags,
>>> +    bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *mapping;
>>> struct amdgpu_bo *bo = bo_va->base.bo;
>>> @@ -1608,7 +1612,7 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>> if (!mapping)
>>> return -ENOMEM;
>>>  -r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size);
>>> +r = amdgpu_vm_bo_clear_mappings(adev, bo_va->base.vm, saddr, size, 
>>> sync_unmap);
>>> if (r) {
>>> kfree(mapping);
>>> return r;
>>> @@ -1633,6 +1637,7 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>>   * @adev: amdgpu_device pointer
>>>   * @bo_va: bo_va to remove the address from
>>>   * @saddr: where to the BO is mapped
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   *
>>>   * Remove a mapping of the BO at the specefied addr from the VM.
>>>   *
>>> @@ -1641,9 +1646,8 @@ int amdgpu_vm_bo_replace_map(struct 
>>> amdgpu_device *adev,
>>>   *
>>>   * Object has to be reserved and unreserved outside!
>>>   */
>>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> -      struct amdgpu_bo_va *bo_va,
>>> -      uint64_t saddr)
>>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> +      uint64_t saddr, bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *mapping;
>>> struct amdgpu_vm *vm = bo_va->base.vm;
>>> @@ -1671,6 +1675,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> list_del(&mapping->list);
>>> amdgpu_vm_it_remove(mapping, &vm->va);
>>> mapping->bo_va = NULL;
>>> +mapping->sync_unmap = sync_unmap;
>>> trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>> if (valid)
>>> @@ -1689,6 +1694,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>>   * @vm: VM structure to use
>>>   * @saddr: start of the range
>>>   * @size: size of the range
>>> + * @sync_unmap: wait for BO users before unmapping
>>>   *
>>>   * Remove all mappings in a range, split them as appropriate.
>>>   *
>>> @@ -1696,8 +1702,8 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>>   * 0 for success, error for failure.
>>>   */
>>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> -struct amdgpu_vm *vm,
>>> -uint64_t saddr, uint64_t size)
>>> +struct amdgpu_vm *vm, uint64_t saddr,
>>> +uint64_t size, bool sync_unmap)
>>>  {
>>> struct amdgpu_bo_va_mapping *before, *after, *tmp, *next;
>>> LIST_HEAD(removed);
>>> @@ -1761,6 +1767,7 @@ int amdgpu_vm_bo_clear_mappings(struct 
>>> amdgpu_device *adev,
>>>    tmp->last = eaddr;
>>> tmp->bo_va = NULL;
>>> +tmp->sync_unmap = sync_unmap;
>>> list_add(&tmp->list, &vm->freed);
>>> trace_amdgpu_vm_bo_unmap(NULL, tmp);
>>> }
>>> @@ -1889,6 +1896,7 @@ void amdgpu_vm_bo_del(struct amdgpu_device *adev,
>>> list_del(&mapping->list);
>>> amdgpu_vm_it_remove(mapping, &vm->va);
>>> mapping->bo_va = NULL;
>>> +mapping->sync_unmap = true;
>>> trace_amdgpu_vm_bo_unmap(bo_va, mapping);
>>> list_add(&mapping->list, &vm->freed);
>>> }
>>> @@ -2617,8 +2625,9 @@ bool amdgpu_vm_handle_fault(struct 
>>> amdgpu_device *adev, u32 pasid,
>>> goto error_unlock;
>>> }
>>>  -r = amdgpu_vm_update_range(adev, vm, true, false, false, NULL, addr,
>>> -  addr, flags, value, 0, NULL, NULL, NULL);
>>> +r = amdgpu_vm_update_range(adev, vm, true, false, false, true, NULL,
>>> +  addr, addr, flags, value, 0, NULL, NULL,
>>> +  NULL);
>>> if (r)
>>> goto error_unlock;
>>>  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index 204ab13184ed..73b7b49fdb2e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -423,12 +423,12 @@ void amdgpu_vm_bo_base_init(struct 
>>> amdgpu_vm_bo_base *base,
>>>    struct amdgpu_vm *vm, struct amdgpu_bo *bo);
>>>  int amdgpu_vm_update_range(struct amdgpu_device *adev, struct 
>>> amdgpu_vm *vm,
>>>   bool immediate, bool unlocked, bool flush_tlb,
>>> -  struct dma_resv *resv, uint64_t start, uint64_t last,
>>> -  uint64_t flags, uint64_t offset, uint64_t vram_base,
>>> +  bool sync_unmap, struct dma_resv *resv,
>>> +  uint64_t start, uint64_t last, uint64_t flags,
>>> +  uint64_t offset, uint64_t vram_base,
>>>   struct ttm_resource *res, dma_addr_t *pages_addr,
>>>   struct dma_fence **fence);
>>> -int amdgpu_vm_bo_update(struct amdgpu_device *adev,
>>> -struct amdgpu_bo_va *bo_va,
>>> +int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> bool clear);
>>>  bool amdgpu_vm_evictable(struct amdgpu_bo *bo);
>>>  void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
>>> @@ -444,15 +444,14 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev,
>>>     uint64_t addr, uint64_t offset,
>>>     uint64_t size, uint64_t flags);
>>>  int amdgpu_vm_bo_replace_map(struct amdgpu_device *adev,
>>> -    struct amdgpu_bo_va *bo_va,
>>> -    uint64_t addr, uint64_t offset,
>>> -    uint64_t size, uint64_t flags);
>>> -int amdgpu_vm_bo_unmap(struct amdgpu_device *adev,
>>> -      struct amdgpu_bo_va *bo_va,
>>> -      uint64_t addr);
>>> +    struct amdgpu_bo_va *bo_va, uint64_t addr,
>>> +    uint64_t offset, uint64_t size, uint64_t flags,
>>> +    bool sync_unmap);
>>> +int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, struct 
>>> amdgpu_bo_va *bo_va,
>>> +      uint64_t addr, bool sync_unmap);
>>>  int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev,
>>> -struct amdgpu_vm *vm,
>>> -uint64_t saddr, uint64_t size);
>>> +struct amdgpu_vm *vm, uint64_t saddr,
>>> +uint64_t size, bool sync_unmap);
>>>  struct amdgpu_bo_va_mapping *amdgpu_vm_bo_lookup_mapping(struct 
>>> amdgpu_vm *vm,
>>> uint64_t addr);
>>>  void amdgpu_vm_bo_trace_cs(struct amdgpu_vm *vm, struct 
>>> ww_acquire_ctx *ticket);
>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c 
>>> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> index bb16b795d1bc..6eb4a0a4bc84 100644
>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>>> @@ -1291,9 +1291,9 @@ svm_range_unmap_from_gpu(struct amdgpu_device 
>>> *adev, struct amdgpu_vm *vm,
>>> pr_debug("[0x%llx 0x%llx]\n", start, last);
>>>  -return amdgpu_vm_update_range(adev, vm, false, true, true, NULL, 
>>> start,
>>> -     last, init_pte_value, 0, 0, NULL, NULL,
>>> -     fence);
>>> +return amdgpu_vm_update_range(adev, vm, false, true, true, true, NULL,
>>> +     start, last, init_pte_value, 0, 0, NULL,
>>> +     NULL, fence);
>>>  }
>>>    static int
>>> @@ -1398,12 +1398,12 @@ svm_range_map_to_gpu(struct 
>>> kfd_process_device *pdd, struct svm_range *prange,
>>> * different memory partition based on fpfn/lpfn, we should use
>>> * same vm_manager.vram_base_offset regardless memory partition.
>>> */
>>> -r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, NULL,
>>> -  last_start, prange->start + i,
>>> -  pte_flags,
>>> -  (last_start - prange->start) << PAGE_SHIFT,
>>> -  bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>>> -  NULL, dma_addr, &vm->last_update);
>>> +r = amdgpu_vm_update_range(
>>> +adev, vm, false, false, flush_tlb, true, NULL,
>>> +last_start, prange->start + i, pte_flags,
>>> +(last_start - prange->start) << PAGE_SHIFT,
>>> +bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
>>> +NULL, dma_addr, &vm->last_update);
>>> for (j = last_start - prange->start; j <= i; j++)
>>> dma_addr[j] |= last_domain;
>>> diff --git a/include/uapi/drm/amdgpu_drm.h 
>>> b/include/uapi/drm/amdgpu_drm.h
>>> index f477eda6a2b8..3cdcc299956e 100644
>>> --- a/include/uapi/drm/amdgpu_drm.h
>>> +++ b/include/uapi/drm/amdgpu_drm.h
>>> @@ -556,6 +556,8 @@ struct drm_amdgpu_gem_op {
>>>  #define AMDGPU_VM_MTYPE_RW(5 << 5)
>>>  /* don't allocate MALL */
>>>  #define AMDGPU_VM_PAGE_NOALLOC(1 << 9)
>>> +/* don't sync on unmap */
>>> +#define AMDGPU_VM_EXPLICIT_SYNC(1 << 10)
>>>    struct drm_amdgpu_gem_va {
>>> /** GEM object handle */
>

[-- Attachment #2: Type: text/html, Size: 85941 bytes --]

[PATCH v2 3/3] drm/amdgpu: Bump amdgpu driver version.

[Tatsuyuki Ishi]

For detection of the new explicit sync functionality without having to try
the ioctl.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 81edf66dbea8..2aa406dee192 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -113,9 +113,10 @@
  *            gl1c_cache_size, gl2c_cache_size, mall_size, enabled_rb_pipes_mask_hi
  *   3.53.0 - Support for GFX11 CP GFX shadowing
  *   3.54.0 - Add AMDGPU_CTX_QUERY2_FLAGS_RESET_IN_PROGRESS support
+ * - 3.55.0 - Add AMDGPU_VM_EXPLICIT_SYNC flag for GEM operations.
  */
 #define KMS_DRIVER_MAJOR	3
-#define KMS_DRIVER_MINOR	54
+#define KMS_DRIVER_MINOR	55
 #define KMS_DRIVER_PATCHLEVEL	0
 
 unsigned int amdgpu_vram_limit = UINT_MAX;
-- 
2.42.0

[PATCH v2 3/3] drm/amdgpu: Bump amdgpu driver version.

[Tatsuyuki Ishi]

For detection of the new explicit sync functionality without having to try
the ioctl.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 81edf66dbea8..2aa406dee192 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -113,9 +113,10 @@
  *            gl1c_cache_size, gl2c_cache_size, mall_size, enabled_rb_pipes_mask_hi
  *   3.53.0 - Support for GFX11 CP GFX shadowing
  *   3.54.0 - Add AMDGPU_CTX_QUERY2_FLAGS_RESET_IN_PROGRESS support
+ * - 3.55.0 - Add AMDGPU_VM_EXPLICIT_SYNC flag for GEM operations.
  */
 #define KMS_DRIVER_MAJOR	3
-#define KMS_DRIVER_MINOR	54
+#define KMS_DRIVER_MINOR	55
 #define KMS_DRIVER_PATCHLEVEL	0
 
 unsigned int amdgpu_vram_limit = UINT_MAX;
-- 
2.42.0