From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AAB96C54E67 for ; Wed, 20 Mar 2024 03:32:23 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 679A610E0A8; Wed, 20 Mar 2024 03:32:23 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="E7AzghW0"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 96F5910E3EE for ; Wed, 20 Mar 2024 03:32:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710905531; x=1742441531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gHYBYFDXKsJgTXQiwTr+Nx8zJYcgL0E03nMuUk9IN+E=; b=E7AzghW0YQutk4pQLh552Zmvfy14rHuyeKbXan5vmV0/mjqt6sXrtwm2 McxuxmJHLR9N6w5ov+oje97frZSrLKgafnNsRd5qymfrjqTyiBpuispIj YEIEubxqlUJOL1Px4UPahYER9koIe2v75W2Ds4131gysA4PsEeikgCJzx cc763+w78Yd+R0uVId6L83yNQlB6v0sBIVAImy1oU5MLRHGCxNUXIkuy/ d/MKaeEVUY1+4F5EwlZ5bA5xz8G1h/rDfvycnHvnbaaRCNUMZQPO3iNWN c81rZyZoJHqOqmxqU9JCt2za3DQPGGUNH1ibU57rlYwBmpnWRP/mqx9WQ g==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5688812" X-IronPort-AV: E=Sophos;i="6.07,138,1708416000"; d="scan'208";a="5688812" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Mar 2024 20:32:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,138,1708416000"; d="scan'208";a="18599115" Received: from szeng-desk.jf.intel.com ([10.165.21.149]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Mar 2024 20:32:10 -0700 From: Oak Zeng To: intel-xe@lists.freedesktop.org Cc: thomas.hellstrom@intel.com, matthew.brost@intel.com, brian.welty@intel.com, himal.prasad.ghimiray@intel.com Subject: [PATCH 8/8] drm/xe: Use hmm_range_fault to populate user pages Date: Tue, 19 Mar 2024 23:44:25 -0400 Message-Id: <20240320034425.1785007-9-oak.zeng@intel.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20240320034425.1785007-1-oak.zeng@intel.com> References: <20240320034425.1785007-1-oak.zeng@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" This is an effort to unify hmmptr (aka system allocator) and userptr code. hmm_range_fault is used to populate a virtual address range for both hmmptr and userptr, instead of hmmptr using hmm_range_fault and userptr using get_user_pages_fast. This also aligns with AMD gpu driver's behavior. In long term, we plan to put some common helpers in this area to drm layer so it can be re-used by different vendors. Signed-off-by: Oak Zeng --- drivers/gpu/drm/xe/xe_vm.c | 122 ++++--------------------------------- 1 file changed, 12 insertions(+), 110 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 51d62323d9ee..11a4bb9d5415 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -38,6 +38,7 @@ #include "xe_sync.h" #include "xe_trace.h" #include "xe_wa.h" +#include "xe_hmm.h" static struct drm_gem_object *xe_vm_obj(struct xe_vm *vm) { @@ -65,113 +66,21 @@ int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma) int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma) { - struct xe_userptr *userptr = &uvma->userptr; struct xe_vma *vma = &uvma->vma; struct xe_vm *vm = xe_vma_vm(vma); struct xe_device *xe = vm->xe; - const unsigned long num_pages = xe_vma_size(vma) >> PAGE_SHIFT; - struct page **pages; - bool in_kthread = !current->mm; - unsigned long notifier_seq; - int pinned, ret, i; - bool read_only = xe_vma_read_only(vma); + struct xe_userptr *userptr; + int ret; lockdep_assert_held(&vm->lock); xe_assert(xe, xe_vma_is_userptr(vma)); -retry: - if (vma->gpuva.flags & XE_VMA_DESTROYED) - return 0; - - notifier_seq = mmu_interval_read_begin(&userptr->notifier); - if (notifier_seq == userptr->notifier_seq) - return 0; - - pages = kvmalloc_array(num_pages, sizeof(*pages), GFP_KERNEL); - if (!pages) - return -ENOMEM; - - if (userptr->sg) { - dma_unmap_sgtable(xe->drm.dev, - userptr->sg, - read_only ? DMA_TO_DEVICE : - DMA_BIDIRECTIONAL, 0); - sg_free_table(userptr->sg); - userptr->sg = NULL; - } - pinned = ret = 0; - if (in_kthread) { - if (!mmget_not_zero(userptr->notifier.mm)) { - ret = -EFAULT; - goto mm_closed; - } - kthread_use_mm(userptr->notifier.mm); - } - - while (pinned < num_pages) { - ret = get_user_pages_fast(xe_vma_userptr(vma) + - pinned * PAGE_SIZE, - num_pages - pinned, - read_only ? 0 : FOLL_WRITE, - &pages[pinned]); - if (ret < 0) - break; + userptr = &uvma->userptr; + mmap_read_lock(userptr->notifier.mm); + ret = xe_userptr_populate_range(uvma); + mmap_read_unlock(userptr->notifier.mm); - pinned += ret; - ret = 0; - } - - if (in_kthread) { - kthread_unuse_mm(userptr->notifier.mm); - mmput(userptr->notifier.mm); - } -mm_closed: - if (ret) - goto out; - - ret = sg_alloc_table_from_pages_segment(&userptr->sgt, pages, - pinned, 0, - (u64)pinned << PAGE_SHIFT, - xe_sg_segment_size(xe->drm.dev), - GFP_KERNEL); - if (ret) { - userptr->sg = NULL; - goto out; - } - userptr->sg = &userptr->sgt; - - ret = dma_map_sgtable(xe->drm.dev, userptr->sg, - read_only ? DMA_TO_DEVICE : - DMA_BIDIRECTIONAL, - DMA_ATTR_SKIP_CPU_SYNC | - DMA_ATTR_NO_KERNEL_MAPPING); - if (ret) { - sg_free_table(userptr->sg); - userptr->sg = NULL; - goto out; - } - - for (i = 0; i < pinned; ++i) { - if (!read_only) { - lock_page(pages[i]); - set_page_dirty(pages[i]); - unlock_page(pages[i]); - } - - mark_page_accessed(pages[i]); - } - -out: - release_pages(pages, pinned); - kvfree(pages); - - if (!(ret < 0)) { - userptr->notifier_seq = notifier_seq; - if (xe_vma_userptr_check_repin(uvma) == -EAGAIN) - goto retry; - } - - return ret < 0 ? ret : 0; + return ret; } static bool preempt_fences_waiting(struct xe_vm *vm) @@ -917,8 +826,6 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, static void xe_vma_destroy_late(struct xe_vma *vma) { struct xe_vm *vm = xe_vma_vm(vma); - struct xe_device *xe = vm->xe; - bool read_only = xe_vma_read_only(vma); if (vma->ufence) { xe_sync_ufence_put(vma->ufence); @@ -926,16 +833,11 @@ static void xe_vma_destroy_late(struct xe_vma *vma) } if (xe_vma_is_userptr(vma)) { - struct xe_userptr *userptr = &to_userptr_vma(vma)->userptr; + struct xe_userptr_vma *uvma = to_userptr_vma(vma); + struct xe_userptr *userptr = &uvma->userptr; - if (userptr->sg) { - dma_unmap_sgtable(xe->drm.dev, - userptr->sg, - read_only ? DMA_TO_DEVICE : - DMA_BIDIRECTIONAL, 0); - sg_free_table(userptr->sg); - userptr->sg = NULL; - } + if (userptr->sg) + xe_userptr_free_sg(uvma); /* * Since userptr pages are not pinned, we can't remove -- 2.26.3