From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A8382C2BB85 for ; Thu, 13 Jun 2024 04:14:32 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 566E410E1F0; Thu, 13 Jun 2024 04:14:32 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="oIB5Xih2"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 14AAB10E210 for ; Thu, 13 Jun 2024 04:13:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1718252035; x=1749788035; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=ZtfDshW3fe5DQhVyl841iwWEqmC+U0aPfmWRPddRwxw=; b=oIB5Xih2IaMuRPP85BjraGozR6djk/2npI8ytsFY1ZlxPAUm4agdxQ0/ KlrKHKkMGSn6i7437nkB4igimFpGBi+b8LWu9Qh0CvZrJpLZH1Agnh4GP Mp5UTK6hdkla0rie79bCEpcuW1jwCYI7fp3za53/bCYklHPzORLYGqqT4 cTinzF+o2R5gbXKMx7MGZakuu52awEfljSWXYiFy67WcS6KUId6/TYW+T GnAjn8FsjNWXhHAleRDKYpW+uqh5WzJHoUbJS7ls+7TqrxjZz3B5N+HO3 Kbd8l2y77a7p2vF2uFEycGIbaX5xaM8HnHaGrl8uUFW87869UkhjDMGyN w==; X-CSE-ConnectionGUID: NrSpqvnCTJ+nhrDnylz4Aw== X-CSE-MsgGUID: LwrJkC0lQ+GwbM2QyhmFTA== X-IronPort-AV: E=McAfee;i="6700,10204,11101"; a="14847996" X-IronPort-AV: E=Sophos;i="6.08,234,1712646000"; d="scan'208";a="14847996" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jun 2024 21:13:49 -0700 X-CSE-ConnectionGUID: ItooO/M9TIqQaLp96UQ7hg== X-CSE-MsgGUID: l5Q4cfuqRzuuPiXJrVN+2A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,234,1712646000"; d="scan'208";a="40476351" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by orviesa006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jun 2024 21:13:48 -0700 From: Oak Zeng To: intel-xe@lists.freedesktop.org Subject: [CI 29/42] drm/xe/svm: Create userptr if page fault occurs on system_allocator VMA Date: Thu, 13 Jun 2024 00:24:16 -0400 Message-Id: <20240613042429.637281-29-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20240613042429.637281-1-oak.zeng@intel.com> References: <20240613042429.637281-1-oak.zeng@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" From: Matthew Brost If a page fault occurs on system_allocator VMA, create a userptr VMA to replaced fault region and map to GPU. v1: Pass userptr to the req_offset of sm_map_ops_create function. This fix malloc'd memory failure (Oak) drop xe_vma_is_userptr(vma) condition. XE_VMA_FAULT_USERPTR is enough to determine a fault userptr (Matt) v2: don't pin fault userptr during fault userptr creation. Fault userptr is created during gpu page fault. After creation, we decide whether we need to migrate it to GPU vram. So pin fault userptr after migration is enough. A pin before migration is redundant. (Oak) v3: remove mm field from xe_vm as we add mm in drm_gpuvm now. rebase (Oak) Cc: Thomas Hellström Cc: Brian Welty Cc: Himal Prasad Ghimiray Signed-off-by: Matthew Brost Signed-off-by: Oak Zeng --- drivers/gpu/drm/xe/xe_gt_pagefault.c | 12 +++ drivers/gpu/drm/xe/xe_vm.c | 118 +++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_vm.h | 7 ++ drivers/gpu/drm/xe/xe_vm_types.h | 1 + 4 files changed, 133 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_gt_pagefault.c b/drivers/gpu/drm/xe/xe_gt_pagefault.c index 3b98499ad614..0bdde3164cd3 100644 --- a/drivers/gpu/drm/xe/xe_gt_pagefault.c +++ b/drivers/gpu/drm/xe/xe_gt_pagefault.c @@ -132,6 +132,18 @@ static int handle_vma_pagefault(struct xe_tile *tile, struct pagefault *pf, trace_xe_vma_pagefault(vma); atomic = access_is_atomic(pf->access_type); + /* + * Create userptr VMA if fault occurs in a range reserved for system + * allocator. + */ + if (xe_vma_is_system_allocator(vma)) { + vma = xe_vm_fault_userptr(vm, pf->page_addr); + if (IS_ERR(vma)) { + xe_vm_kill(vm, true); + return PTR_ERR(vma); + } + } + retry_userptr: if (xe_vma_is_userptr(vma) && xe_vma_userptr_check_repin(to_userptr_vma(vma))) { diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 499b95f6422b..dba7a64075e9 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -1482,7 +1482,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags) } drm_gpuvm_init(&vm->gpuvm, "Xe VM", DRM_GPUVM_RESV_PROTECTED, &xe->drm, - vm_resv_obj, 0, vm->size, 0, 0, &gpuvm_ops, false); + vm_resv_obj, 0, vm->size, 0, 0, &gpuvm_ops, + flags & XE_VM_FLAG_PARTICIPATE_SVM); drm_gem_object_put(vm_resv_obj); @@ -2093,7 +2094,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, if (bo) drm_exec_fini(&exec); - if (xe_vma_is_userptr(vma)) { + if (xe_vma_is_userptr(vma) && !xe_vma_is_fault_userptr(vma)) { err = xe_vma_userptr_pin_pages(to_userptr_vma(vma)); if (err) { prep_vma_destroy(vm, vma, false); @@ -2207,8 +2208,11 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op) return err; } -static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops, - struct xe_vma_ops *vops) +static int vm_bind_ioctl_ops_update_gpuvm_state(struct xe_vm *vm, + struct drm_gpuva_ops *ops, + struct xe_sync_entry *syncs, + u32 num_syncs, + struct xe_vma_ops *vops) { struct xe_device *xe = vm->xe; struct drm_gpuva_op *__op; @@ -3145,7 +3149,8 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto unwind_ops; } - err = vm_bind_ioctl_ops_parse(vm, ops[i], &vops); + err = vm_bind_ioctl_ops_update_gpuvm_state(vm, ops[i], syncs, + num_syncs, &vops); if (err) goto unwind_ops; } @@ -3464,3 +3469,106 @@ void xe_vm_snapshot_free(struct xe_vm_snapshot *snap) } kvfree(snap); } + +/** + * xe_vm_fault_userptr() - VM fault userptr + * @vm: VM + * @fault_addr: fault address + * + * Create userptr VMA from fault address + * + * Return: newly created userptr VMA on success, ERR_PTR on failure + */ +struct xe_vma *xe_vm_fault_userptr(struct xe_vm *vm, u64 fault_addr) +{ + struct vm_area_struct *vas; + struct mm_struct *mm = vm->gpuvm.mm; + struct xe_vma_ops vops; + struct drm_gpuva_ops *ops = NULL; + struct drm_gpuva_op *__op; + struct xe_vma *vma = NULL; + u64 start, range; + int err; + + vm_dbg(&vm->xe->drm, "FAULT: addr=0x%016llx", fault_addr); + + if (!mmget_not_zero(mm)) + return ERR_PTR(-EFAULT); + + kthread_use_mm(mm); + + mmap_read_lock(mm); + vas = find_vma_intersection(mm, fault_addr, fault_addr + 4); + if (!vas) { + err = -ENOENT; + goto err_unlock; + } + + vm_dbg(&vm->xe->drm, "FOUND VAS: vm_start=0x%016lx, vm_end=0x%016lx", + vas->vm_start, vas->vm_end); + + start = vas->vm_start; + range = vas->vm_end - vas->vm_start; + mmap_read_unlock(mm); + + ops = drm_gpuvm_sm_map_ops_create(&vm->gpuvm, start, range, 0, start); + if (IS_ERR(ops)) { + err = PTR_ERR(ops); + goto err_kthread; + } + + drm_gpuva_for_each_op(__op, ops) + print_op(vm->xe, __op); + + xe_vma_ops_init(&vops, vm, NULL, NULL, 0); + err = vm_bind_ioctl_ops_update_gpuvm_state(vm, ops, NULL, 0, &vops); + if (err) + goto err_kthread; + + /* + * No need to execute ops as we just want to update GPUVM state, page + * fault handler will update GPU page tables. Find VMA that needs GPU + * mapping and return to page fault handler. + */ + xe_vm_lock(vm, false); + drm_gpuva_for_each_op(__op, ops) { + struct xe_vma_op *op = gpuva_op_to_vma_op(__op); + + if (__op->op == DRM_GPUVA_OP_MAP) { + xe_assert(vm->xe, !vma); + vma = op->map.vma; + vma->gpuva.flags |= XE_VMA_FAULT_USERPTR; + } else if (__op->op == DRM_GPUVA_OP_UNMAP) { + xe_vma_destroy(gpuva_to_vma(op->base.unmap.va), NULL); + } else if (__op->op == DRM_GPUVA_OP_REMAP) { + xe_vma_destroy(gpuva_to_vma(op->base.remap.unmap->va), + NULL); + } + } + xe_vm_unlock(vm); + + kthread_unuse_mm(mm); + mmput(mm); + drm_gpuva_ops_free(&vm->gpuvm, ops); + + return vma; + +err_unlock: + mmap_read_unlock(mm); +err_kthread: + kthread_unuse_mm(mm); + mmput(mm); + if (ops) { + drm_gpuva_for_each_op_reverse(__op, ops) { + struct xe_vma_op *op = gpuva_op_to_vma_op(__op); + + xe_vma_op_unwind(vm, op, + op->flags & XE_VMA_OP_COMMITTED, + op->flags & XE_VMA_OP_PREV_COMMITTED, + op->flags & XE_VMA_OP_NEXT_COMMITTED); + } + drm_gpuva_ops_free(&vm->gpuvm, ops); + } + + return ERR_PTR(err); +} diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h index 680c0c49b2f4..a31409b87b8a 100644 --- a/drivers/gpu/drm/xe/xe_vm.h +++ b/drivers/gpu/drm/xe/xe_vm.h @@ -172,6 +172,11 @@ static inline bool xe_vma_is_userptr(struct xe_vma *vma) !xe_vma_is_system_allocator(vma); } +static inline bool xe_vma_is_fault_userptr(struct xe_vma *vma) +{ + return vma->gpuva.flags & XE_VMA_FAULT_USERPTR; +} + /** * to_userptr_vma() - Return a pointer to an embedding userptr vma * @vma: Pointer to the embedded struct xe_vma @@ -252,6 +257,8 @@ int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma); int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma); +struct xe_vma *xe_vm_fault_userptr(struct xe_vm *vm, u64 fault_addr); + bool xe_vm_validate_should_retry(struct drm_exec *exec, int err, ktime_t *end); int xe_vm_lock_vma(struct drm_exec *exec, struct xe_vma *vma); diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h index 165c2ca258de..c1bffa60cefc 100644 --- a/drivers/gpu/drm/xe/xe_vm_types.h +++ b/drivers/gpu/drm/xe/xe_vm_types.h @@ -33,6 +33,7 @@ struct xe_vm_pgtable_update_op; #define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 7) #define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 8) #define XE_VMA_SYSTEM_ALLOCATOR (DRM_GPUVA_USERBITS << 9) +#define XE_VMA_FAULT_USERPTR (DRM_GPUVA_USERBITS << 10) /** struct xe_userptr - User pointer */ struct xe_userptr { -- 2.26.3