From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC34EFF8862 for ; Mon, 27 Apr 2026 10:06:06 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 44D3710E689; Mon, 27 Apr 2026 10:06:06 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (1024-bit key; unprotected) header.d=amd.com header.i=@amd.com header.b="STUr9PH1"; dkim-atps=neutral Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012004.outbound.protection.outlook.com [40.107.209.4]) by gabe.freedesktop.org (Postfix) with ESMTPS id B9C2310E690; Mon, 27 Apr 2026 10:06:04 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BUjR+g46HpMHvNgc+gE3OcDS6RArxiLRDCAX8DcP4tK6X0lU/qGHnLAo+I+awFHA9V9Z6/afNFIRR13aNOl4k0S2hjssTszqdX0VIVnqinJRVw0+auYl8UXHklLCA7s07opgG0WaNwb42TOcNZDmQ/GWUeTSi+V7tjlQ8McFqtFJAF7bbzQl1SrzNqM9Kj4MFJZSzNJDSegLBuODIyl0XI5zFXmMzRR107CGiTTmUo/YUTO0sO11nTx6w6ShzsRHwm9TroKO+eVxxbA9okZDiVl+uVOr79AbcP+X7o4a3hrTCBjzGmkSS8V+TaygUjLAmePYzhKmhjuvFoBB0OQ+gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AjhknTxeRN89/7QX/XYa5NGVLUM7Hj0pazD2+B38Q+c=; b=wWmWuQmOztn6l307GWuRFNIArSko2+6VuCX2XAU4xkQqpArNJwXur21OqJwqKjzw7j0uETVSNV/a3SHTkU4YMe+w0gFywTGB/KQm0r8f7hmp5bXY/Fx8ZaY+MM4elqKWgVU9d20jyNfEqDBUF9hQbZ7OBg4Y+tIH4Z+64pyUQMNJInAErOFt5oAsiamsLxnQ8n2bsUGNpBBe6hO1f+p5yilmKjmsrbv46fIxQBWPlkwwQFQYfDdS3Z8xjXsJXEBkCua3tX7v3j4oY/8zB7X6aoTrJRmDWMLyMj0/FgTtQWqcAXZrvRj0VSm3iKvw+jMjPtbaonl0jV1VOplGMzwvtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AjhknTxeRN89/7QX/XYa5NGVLUM7Hj0pazD2+B38Q+c=; b=STUr9PH1lyTdC2TOQW4V/RSUAolD6kBJ9gZqPI8p8K70FBake9fmb1OVYibyQ1ApF54CaJ8KU2h29ffnOK6KUjuFvWotlgAK6Nvg0BHkEXpnjHJEJAS7xnbe2ZLR0Jwi7KwcvBeAodxBVN+HJYFGMiWCqwNF50FlJEsVUqn7B5o= Received: from SJ0PR03CA0085.namprd03.prod.outlook.com (2603:10b6:a03:331::30) by SN7PR12MB8770.namprd12.prod.outlook.com (2603:10b6:806:34b::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.15; Mon, 27 Apr 2026 10:05:46 +0000 Received: from MW1PEPF0001615E.namprd21.prod.outlook.com (2603:10b6:a03:331:cafe::25) by SJ0PR03CA0085.outlook.office365.com (2603:10b6:a03:331::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9846.26 via Frontend Transport; Mon, 27 Apr 2026 10:05:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by MW1PEPF0001615E.mail.protection.outlook.com (10.167.249.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9891.0 via Frontend Transport; Mon, 27 Apr 2026 10:05:45 +0000 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 27 Apr 2026 05:05:37 -0500 Received: from junhua-PC.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Mon, 27 Apr 2026 05:05:34 -0500 From: Junhua Shen To: , , , , , , , , , CC: , , "Junhua Shen" Subject: [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks Date: Mon, 27 Apr 2026 18:05:19 +0800 Message-ID: <20260427100522.7014-3-Junhua.Shen@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260427100522.7014-1-Junhua.Shen@amd.com> References: <20260427100522.7014-1-Junhua.Shen@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MW1PEPF0001615E:EE_|SN7PR12MB8770:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e614460-644d-48b5-e1ef-08dea4448cba X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|36860700016|376014|1800799024|56012099003|22082099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: cJOiCi59+yVgMjU4bVmIN3vghbWxXIA8ZAyE6TX4kzT6b1YFwZDNGlLrrUzLYnPvtuOqROQgUP9DNwAi7kIhlu7kj6RV/R2f8K465FwpDK44s3JXjHcjMDnnAUi26cCSF9Nn7RASdwMd925DOodpsiK7hcUrieNOE8N4r7tE9QyiNAB2ago3niHd6/zSBhbsE4QZGBLaEbzQ1TSQO0pQzsa+/Ekq26UckunhJDSVjSyTSyGZWCSDSjjYxpUZNZ7TYo7CfI+9NuXHFzNhYKIgvjRz0ySsnXbiRSCCmHMvyWNPJbb2oMprutfoY+Ll6Ox7Av7gElfftkx688Iv+0FNAsP5Irm+Ir7Lr/B7N/9f6apoUJ+ow4729slw9eFRZe4fwKdIN5LA9hUJ+I9U3m7CUhFaZBu/YEEhEohQQ6YOEk1XfMGeqJHZbe/mu1QHGN3bN3weXvOg4Nyw/t34sK7eHXIeDj4WU2d/V5wTZniV2L9gs62/d1YugmeE/nVOijLc8iV2n0ZFLiSmE+z5gWDc5q5oQ7tFvvY5m+Vkf42UxjUuSQ0g6OKsyiGmJYWwnwI7yrKr5bJH/JRz2Hx5mfLkRMmpAKCegyRcS5sHOeVPB+cjgzhDrzIrjsaNWx7glTr+UceuBAczqvtxb55tAg+MTlVhUrS2Xdcan56BheSzbIfAIWB3NWIGbtaoK1cbpdIfdBQ/pk38Iuluw/r85nw+Gp4UT0rvd0APrhO8snbYoXwcFoiHWpz9Wx+S0aVoZIZVCZbs+oxYZlv9flF6nFAS3gTvhcWS5GGqECarKB+c0Gg= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:satlexmb08.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(82310400026)(36860700016)(376014)(1800799024)(56012099003)(22082099003)(18002099003)(921020); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: nfgCsZKTUS3Z+JCNTjagm5T/aZRzLKJvzzw3o/Z49YvF5N7iyBZdu7L58DjG0WTYi3idGhhucpRbodjw9B010AL1X7VwetoMVDbVBf/hMNZYvKrlTDDiybLmWCatPNx1iBVkTdZAA8Ra7K6qRONvrtzsRNbbLlxuqS/FPsIZXu1wu2Qb9Bu+0tQWNiP7eJ3XKIHo1DwHF5HRwEAa7hScxZponm0qQV5XSbS82YjQy0Vv8j5iDxvyXDmNQh3s2JSygiEvLc10NWEi/NAjwrfQpCLL8zy5CXGta0Eu4FlR7AdFLnK0CJJUbJ3xY2x6Ei2CrZ3naK76FSfWGUJhzE2BUPBlS8yKTp+RCHKhqY9Dh9z2Od/TLrwt7rw6S7jqAk9pE+MJ9VHjcLbx27DsafIe+yzktbNtJTHBRO5oG8BUlzsQXIEpvMurbgliYZ+000B8 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2026 10:05:45.8226 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6e614460-644d-48b5-e1ef-08dea4448cba X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MW1PEPF0001615E.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8770 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks that the DRM GPUSVM migration framework requires: drm_pagemap_ops (top-level entry points): - device_map: convert ZONE_DEVICE page to GPU PTE address - populate_mm: allocate VRAM BO and trigger migration drm_pagemap_devmem_ops (per-BO migration mechanics): - populate_devmem_pfn: walk BO buddy blocks to build PFN array - copy_to_devmem: SDMA copy system RAM -> VRAM via GART window - copy_to_ram: SDMA copy VRAM -> system RAM via GART window - devmem_release: free BO when all pages migrate back Signed-off-by: Junhua Shen --- drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 616 +++++++++++++++++++- 1 file changed, 613 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c index 170e2eadc106..42092651b4d5 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c @@ -64,12 +64,20 @@ #include #include -#include "amdgpu_amdkfd.h" #include "amdgpu_migrate.h" #include "amdgpu.h" +#include "amdgpu_ttm.h" +#include "amdgpu_res_cursor.h" + +#define AMDGPU_MIGRATE_TRACE(fmt, ...) \ + pr_debug("%s: " fmt, __func__, ##__VA_ARGS__) + +/* SDMA copy direction */ +#define FROM_RAM_TO_VRAM 0 +#define FROM_VRAM_TO_RAM 1 static inline struct amdgpu_pagemap * -dpagemap_to_apagemap(struct drm_pagemap *dpagemap) +to_amdgpu_pagemap(struct drm_pagemap *dpagemap) { return container_of(dpagemap, struct amdgpu_pagemap, dpagemap); } @@ -94,8 +102,610 @@ amdgpu_svm_page_to_apagemap(struct page *page) return container_of(pgmap, struct amdgpu_pagemap, pgmap); } +/* drm_pagemap_devmem_ops — per-BO migration mechanics */ + +/** + * struct amdgpu_svm_bo - Wrapper linking drm_pagemap_devmem to amdgpu_bo + * + * @devmem: drm_pagemap device memory allocation (passed to framework) + * @bo: The backing VRAM amdgpu_bo + * + * It is allocated per-migration in populate_mm() and freed by + * devmem_release() when all device-private pages have migrated + * back to system memory. + * + * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount: + * - zdd->devmem_allocation points to &svm_bo->devmem + * - When zdd refcount drops to zero, framework calls devmem_release() + * - devmem_release() frees both the BO reference and the svm_bo itself + */ +struct amdgpu_svm_bo { + struct amdgpu_bo *bo; + struct drm_pagemap_devmem devmem; +}; + +static inline struct amdgpu_svm_bo * +to_amdgpu_svm_bo(struct drm_pagemap_devmem *devmem_allocation) +{ + return container_of(devmem_allocation, struct amdgpu_svm_bo, devmem); +} + +/** + * amdgpu_svm_devmem_release - Release BO when all device pages migrate back + * + * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the + * last device-private page backed by this allocation has been migrated back + * to system memory (or the owning process exits). + * + * Frees both the amdgpu_bo reference and the wrapper amdgpu_svm_bo itself. + */ +static void +amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation) +{ + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation); + + AMDGPU_MIGRATE_TRACE("Release svm_bo=%px bo=%px\n", svm_bo, svm_bo->bo); + amdgpu_bo_unref(&svm_bo->bo); + kfree(svm_bo); +} + +/** + * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array + * @devmem_allocation: The devmem allocation in the amdgpu_svm_bo wrapper + * @npages: Number of PFN entries to fill + * @pfn: Output PFN array + * + * Iterates over the BO's TTM vram_mgr buddy blocks and converts each + * block's VRAM offset to ZONE_DEVICE PFNs: + * + * PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index + * + * This is called by drm_pagemap_migrate_to_devmem() to build the + * destination PFN array for migrate_vma_pages(). + * + * Return: 0 on success + */ +static int +amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation, + unsigned long npages, unsigned long *pfn) +{ + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(devmem_allocation->dpagemap); + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation); + struct amdgpu_bo *bo = svm_bo->bo; + struct amdgpu_res_cursor cursor; + unsigned long i = 0; + int ret; + + ret = amdgpu_bo_reserve(bo, false); + if (ret) + return ret; + + amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor); + while (cursor.remaining && i < npages) { + u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base); + u64 pages = cursor.size >> PAGE_SHIFT; + unsigned long j; + + for (j = 0; j < pages && i < npages; j++, i++) + pfn[i] = pfn_base + j; + + amdgpu_res_next(&cursor, cursor.size); + } + + amdgpu_bo_unreserve(bo); + + AMDGPU_MIGRATE_TRACE("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n", + npages, npages > 0 ? pfn[0] : 0); + + return 0; +} + +/* SDMA copy helpers — GART window based data transfer */ + +/** + * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address + * @adev: AMDGPU device + * @vram_offset: Byte offset within VRAM + * + * Return: MC address suitable for SDMA src/dst + */ +static u64 +amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset) +{ + return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM); +} + +/** + * amdgpu_svm_gart_map - Map system DMA addresses into GART window + * @ring: SDMA ring for the GART update job + * @npages: Number of pages to map + * @addr: Array of system memory DMA addresses + * @gart_addr: Output — GART base address to use in SDMA copy + * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src) + * + * Builds GART PTEs pointing at the given DMA addresses, submits an + * SDMA job to update the GART entries, and returns the GART address + * that can be used as src or dst in a subsequent amdgpu_copy_buffer(). + * + * Uses GART window 0, protected by gtt_window_lock. + * + * Return: 0 on success, negative error code on failure + */ +static int +amdgpu_svm_gart_map(struct amdgpu_ring *ring, + struct amdgpu_ttm_buffer_entity *entity, + u64 npages, + dma_addr_t *addr, u64 *gart_addr, u64 flags) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_job *job; + unsigned int num_dw, num_bytes; + struct dma_fence *fence; + u64 src_addr, dst_addr; + u64 pte_flags; + void *cpu_addr; + int r; + + /* Use entity's GART window 0 */ + *gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0); + + num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8); + num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE; + + r = amdgpu_job_alloc_with_ib(adev, &entity->base, + AMDGPU_FENCE_OWNER_UNDEFINED, + num_dw * 4 + num_bytes, + AMDGPU_IB_POOL_DELAYED, + &job, + AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP); + if (r) + return r; + + src_addr = num_dw * 4; + src_addr += job->ibs[0].gpu_addr; + + dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo); + dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8; + amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr, + dst_addr, num_bytes, 0); + + amdgpu_ring_pad_ib(ring, &job->ibs[0]); + WARN_ON(job->ibs[0].length_dw > num_dw); + + pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE; + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; + if (flags & AMDGPU_PTE_WRITEABLE) + pte_flags |= AMDGPU_PTE_WRITEABLE; + pte_flags |= adev->gart.gart_pte_flags; + + cpu_addr = &job->ibs[0].ptr[num_dw]; + + amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr); + fence = amdgpu_job_submit(job); + dma_fence_put(fence); + + return 0; +} + +/** + * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM + * @adev: AMDGPU device + * @sys: Array of DMA addresses for system memory pages + * @vram: Array of VRAM byte offsets (relative to start of VRAM) + * @npages: Number of pages to copy + * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM + * @mfence: In/out — carries the last SDMA fence for serialization + * + * Maps system memory pages into the GART window and uses SDMA to copy + * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE + * chunks. Acquires entity->lock internally to protect the GART window, + * matching the KFD svm_migrate_copy_memory_gart() pattern. + * + * Return: 0 on success, negative error code on failure + */ +static int +amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys, + u64 *vram, u64 npages, int direction, + struct dma_fence **mfence) +{ + const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE; + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; + struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entity; + u64 gart_s, gart_d; + struct dma_fence *next; + u64 size; + int r; + + mutex_lock(&entity->lock); + + while (npages) { + size = min(max_pages, npages); + + if (direction == FROM_VRAM_TO_RAM) { + gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram); + r = amdgpu_svm_gart_map(ring, entity, size, sys, + &gart_d, AMDGPU_PTE_WRITEABLE); + } else { + r = amdgpu_svm_gart_map(ring, entity, size, sys, + &gart_s, 0); + gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram); + } + if (r) { + dev_err(adev->dev, "failed %d to map GART for SDMA\n", r); + goto out_unlock; + } + + AMDGPU_MIGRATE_TRACE("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n", + direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM", + size, (u64)*vram); + + r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d, + size * PAGE_SIZE, + NULL, &next, true, 0); + if (r) { + dev_err(adev->dev, "failed %d to copy buffer\n", r); + goto out_unlock; + } + + dma_fence_put(*mfence); + *mfence = next; + npages -= size; + if (npages) { + sys += size; + vram += size; + } + } + +out_unlock: + mutex_unlock(&entity->lock); + + return r; +} + +/** + * amdgpu_svm_copy_to_devmem - SDMA copy system memory -> VRAM + * @pages: Array of destination ZONE_DEVICE pages (VRAM-backed) + * @pagemap_addr: Array of source DMA addresses (system memory, already mapped) + * @npages: Number of pages to copy + * + * Builds parallel sys[] and vram[] arrays from the framework-provided + * pagemap_addr and device pages, then submits batched SDMA copies via + * the GART window. + * + * Return: 0 on success, negative error code on failure + */ +static int +amdgpu_svm_copy_to_devmem(struct page **pages, + struct drm_pagemap_addr *pagemap_addr, + unsigned long npages, + struct dma_fence *pre_migrate_fence) +{ + struct amdgpu_device *adev; + struct amdgpu_pagemap *svm_dm; + struct dma_fence *mfence = NULL; + dma_addr_t *sys; + u64 *vram; + unsigned long i, j; + int ret = 0; + + if (!npages) + return 0; + + /* + * Find the first non-NULL page to derive the device. + * The pages array may contain NULL entries for positions where + * no valid device page exists. + */ + for (i = 0; i < npages; i++) { + if (pages[i]) + break; + } + if (i == npages) + return 0; + + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]); + adev = svm_dm->adev; + + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL); + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL); + if (!sys || !vram) { + ret = -ENOMEM; + goto out_free; + } + + for (i = 0, j = 0; i < npages; i++) { + if (!pagemap_addr[i].addr) + goto flush; + + sys[j] = pagemap_addr[i].addr; + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) - + svm_dm->hpa_base; + + /* Check if next vram page is contiguous with current */ + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE) + goto flush; + + j++; + continue; +flush: + if (j) { + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j, + FROM_RAM_TO_VRAM, + &mfence); + if (ret) + goto out_fence; + j = 0; + } + /* Re-process current page if it was valid but broke contiguity */ + if (pagemap_addr[i].addr) { + sys[0] = pagemap_addr[i].addr; + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) - + svm_dm->hpa_base; + j = 1; + } + } + + /* Flush remaining batch */ + if (j) + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j, + FROM_RAM_TO_VRAM, &mfence); + +out_fence: + if (mfence) { + dma_fence_wait(mfence, false); + dma_fence_put(mfence); + } + + AMDGPU_MIGRATE_TRACE("copy_to_devmem done: npages=%ld ret=%d\n", + npages, ret); + +out_free: + kvfree(vram); + kvfree(sys); + return ret; +} + +/** + * amdgpu_svm_copy_to_ram - SDMA copy VRAM -> system memory + * @pages: Array of source ZONE_DEVICE pages (VRAM-backed) + * @pagemap_addr: Array of destination DMA addresses (system memory, already mapped) + * @npages: Number of pages to copy + * + * Mirror of copy_to_devmem with src/dst swapped. + * + * Return: 0 on success, negative error code on failure + */ +static int +amdgpu_svm_copy_to_ram(struct page **pages, + struct drm_pagemap_addr *pagemap_addr, + unsigned long npages, + struct dma_fence *pre_migrate_fence) +{ + struct amdgpu_device *adev; + struct amdgpu_pagemap *svm_dm; + struct dma_fence *mfence = NULL; + dma_addr_t *sys; + u64 *vram; + unsigned long i, j; + int ret = 0; + + if (!npages) + return 0; + + for (i = 0; i < npages; i++) { + if (pages[i]) + break; + } + if (i == npages) + return 0; + + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]); + adev = svm_dm->adev; + + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL); + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL); + if (!sys || !vram) { + ret = -ENOMEM; + goto out_free; + } + + for (i = 0, j = 0; i < npages; i++) { + if (!pagemap_addr[i].addr || !pages[i]) + goto flush; + + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) - + svm_dm->hpa_base; + sys[j] = pagemap_addr[i].addr; + + /* Check if next vram page is contiguous with current */ + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE) + goto flush; + + j++; + continue; +flush: + if (j) { + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j, + FROM_VRAM_TO_RAM, + &mfence); + if (ret) + goto out_fence; + j = 0; + } + /* Re-process current page if it was valid but broke contiguity */ + if (pagemap_addr[i].addr && pages[i]) { + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) - + svm_dm->hpa_base; + sys[0] = pagemap_addr[i].addr; + j = 1; + } + } + + /* Flush remaining batch */ + if (j) + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j, + FROM_VRAM_TO_RAM, &mfence); + +out_fence: + if (mfence) { + dma_fence_wait(mfence, false); + dma_fence_put(mfence); + } + + AMDGPU_MIGRATE_TRACE("copy_to_ram done: npages=%ld ret=%d\n", npages, ret); + +out_free: + kvfree(vram); + kvfree(sys); + return ret; +} + +static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = { + .devmem_release = amdgpu_svm_devmem_release, + .populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn, + .copy_to_devmem = amdgpu_svm_copy_to_devmem, + .copy_to_ram = amdgpu_svm_copy_to_ram, +}; + +/* drm_pagemap_ops — top-level migration entry points */ + +/** + * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address + * @dpagemap: The drm_pagemap for this device + * @dev: Requesting device (for P2P check) + * @page: ZONE_DEVICE page backed by VRAM + * @order: Page order (0 = 4K, 9 = 2M, etc.) + * @dir: DMA direction (unused for local VRAM) + * + * Address conversion chain: + * page -> PFN -> HPA -> VRAM offset -> PTE address + * + * HPA = page_to_pfn(page) << PAGE_SHIFT + * VRAM offset = HPA - apagemap.hpa_base + * PTE address = VRAM offset + adev->vm_manager.vram_base_offset + * + * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol + */ +static struct drm_pagemap_addr +amdgpu_svm_device_map(struct drm_pagemap *dpagemap, + struct device *dev, + struct page *page, + unsigned int order, + enum dma_data_direction dir) +{ + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(dpagemap); + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap); + dma_addr_t addr; + + if (dpagemap->drm->dev == dev) { + /* Same device: return VRAM PTE address */ + u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT; + u64 vram_offset = hpa - svm_dm->hpa_base; + + addr = vram_offset + adev->vm_manager.vram_base_offset; + } else { + /* Cross-device P2P: not yet supported */ + addr = DMA_MAPPING_ERROR; + } + + return drm_pagemap_addr_encode(addr, + AMDGPU_INTERCONNECT_VRAM, order, dir); +} + +/** + * amdgpu_svm_bo_alloc - Allocate an amdgpu_svm_bo wrapper with VRAM backing + * @adev: AMDGPU device + * @dpagemap: The drm_pagemap for this device + * @mm: mm_struct of the owning process + * @size: Allocation size in bytes + * + * Return: Pointer to allocated amdgpu_svm_bo on success, ERR_PTR on failure + */ +static struct amdgpu_svm_bo * +amdgpu_svm_bo_alloc(struct amdgpu_device *adev, + struct drm_pagemap *dpagemap, + struct mm_struct *mm, unsigned long size) +{ + struct amdgpu_svm_bo *svm_bo; + struct amdgpu_bo_param bp = {}; + struct amdgpu_bo *bo; + int ret; + + svm_bo = kzalloc(sizeof(*svm_bo), GFP_KERNEL); + if (!svm_bo) + return ERR_PTR(-ENOMEM); + + bp.size = size; + bp.bo_ptr_size = sizeof(struct amdgpu_bo); + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.type = ttm_bo_type_device; + bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS | + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS | + AMDGPU_GEM_CREATE_VRAM_CLEARED; + + ret = amdgpu_bo_create(adev, &bp, &bo); + if (ret) { + AMDGPU_MIGRATE_TRACE("Failed to create SVM BO\n"); + kfree(svm_bo); + return ERR_PTR(ret); + } + + amdgpu_bo_unreserve(bo); + svm_bo->bo = bo; + + drm_pagemap_devmem_init(&svm_bo->devmem, + adev->dev, mm, + &amdgpu_pagemap_ops, + dpagemap, size, NULL); + + return svm_bo; +} + +/** + * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages + * @dpagemap: The drm_pagemap for this device + * @start: Start virtual address of the range to migrate + * @end: End virtual address (exclusive) + * @mm: mm_struct of the owning process + * @timeslice_ms: Maximum time to spend migrating (for fairness) + * + * Core migration entry point called by drm_pagemap_populate_mm(). + * Allocates an amdgpu_svm_bo via amdgpu_svm_bo_alloc(), then calls + * drm_pagemap_migrate_to_devmem() to execute the actual migration. + * + * Return: 0 on success, negative error code on failure + */ +static int +amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap, + unsigned long start, unsigned long end, + struct mm_struct *mm, + unsigned long timeslice_ms) +{ + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap); + struct drm_pagemap_migrate_details mdetails = { + .timeslice_ms = timeslice_ms, + }; + struct amdgpu_svm_bo *svm_bo; + int ret; + + svm_bo = amdgpu_svm_bo_alloc(adev, dpagemap, mm, end - start); + if (IS_ERR(svm_bo)) + return PTR_ERR(svm_bo); + + AMDGPU_MIGRATE_TRACE("populate_mm: [0x%lx-0x%lx] size=%lu\n", + start, end, end - start); + + ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem, + mm, start, end, + &mdetails); + + return ret; +} -const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { }; +const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { + .device_map = amdgpu_svm_device_map, + .populate_mm = amdgpu_svm_populate_mm, +}; /** * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap -- 2.34.1