From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9CA8C48BF6 for ; Thu, 29 Feb 2024 09:41:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5C96E10E34C; Thu, 29 Feb 2024 09:41:39 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (1024-bit key; unprotected) header.d=amd.com header.i=@amd.com header.b="S3JHCsjM"; dkim-atps=neutral Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2046.outbound.protection.outlook.com [40.107.95.46]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0296810E34C; Thu, 29 Feb 2024 09:41:37 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hrq4jIfljETosXtZJXeA69IYDNYZPF0oFhBuSuttN6PBNa7M7ec2eGF8W21OZE97KYNihxFWb6M2Ylzu5FwmlvFeMEUiWRAnhQtG4FJUbO3VrmMuwsnMstUFX7VGOTeU7pfH3zwijccECfpfRi2Xjyh5BvT+pz3fx98O/nWdFXdqUgKpnafcUVSQKO38AUcb8laF0JX6f9TyVSebSx5CeNOR//DpvszP0YZolEAZTB4XCKW9kcVcIQsyfkkVOTLF6DMiK/W3CxC9T/J5q/hL3gan8ikG/r3iGVblJtzK71OSwy9s2Bld2Xt9ne0z2Ej7Lh53DPQgkdyHqSQ+z3qCWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jWYX3h7ebScijFeL3Qo7T/ZPD0WZGQlg1ieNzTKePWA=; b=XfDqItrrhQwMzO/X3a/STW7DxPyyP7IS0TZmWD7BsIhNydSOMxzoVr4rju2gyCnohHCi9nrSIz/dBcDiXklZzJL/wcuOlJt+HH4j7s8ZtoUYEKPHoesBs7Y21rXxw9Lw6JQpG+eRuJyl74xe1u6+4ubaSOiuDrNjnvrhqOkniPBKnQNwNW39iiiI/XrimlSzrKDvwFLIj7niu9cUj04FexMtPXjHfmDz21WLUajjsGGJTfTVTR7dAl2fFbaau0n74CIOcAGGWC+msvfeRoVZs0mMgSQyUBroGyIV+97ULRHT2bXCLhX6LAjYN4E63Fs6O8//sWk2qUdkymdhR7sIlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jWYX3h7ebScijFeL3Qo7T/ZPD0WZGQlg1ieNzTKePWA=; b=S3JHCsjMIX66dP5z/AgMCL2UneOTab+OEKgSmcdkGtHZnzzn3MNi8ppiCh4coYScM+jTpe5+MG2zgS8CBF/GVh11noXQ/F2M/rE6djcezeBRNQtklkkOWCK1MOOCznnzT1HpOfmClOjX1qQvsDNlZ9Ftd0WSR5yQmefg/2LXbKY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from PH7PR12MB5685.namprd12.prod.outlook.com (2603:10b6:510:13c::22) by CH3PR12MB8726.namprd12.prod.outlook.com (2603:10b6:610:17b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.41; Thu, 29 Feb 2024 09:41:30 +0000 Received: from PH7PR12MB5685.namprd12.prod.outlook.com ([fe80::f2b6:1034:76e8:f15a]) by PH7PR12MB5685.namprd12.prod.outlook.com ([fe80::f2b6:1034:76e8:f15a%6]) with mapi id 15.20.7316.031; Thu, 29 Feb 2024 09:41:30 +0000 Content-Type: multipart/alternative; boundary="------------6thb4d74p0ITt3oJxb4h3ERf" Message-ID: Date: Thu, 29 Feb 2024 10:41:24 +0100 User-Agent: Mozilla Thunderbird Subject: Re: Making drm_gpuvm work across gpu devices Content-Language: en-US To: "Zeng, Oak" , Danilo Krummrich , Dave Airlie , Daniel Vetter , Felix Kuehling , "jglisse@redhat.com" Cc: "Welty, Brian" , "dri-devel@lists.freedesktop.org" , "intel-xe@lists.freedesktop.org" , "Bommu, Krishnaiah" , "Ghimiray, Himal Prasad" , "Thomas.Hellstrom@linux.intel.com" , "Vishwanathapura, Niranjana" , "Brost, Matthew" , "Gupta, saurabhg" References: <20240117221223.18540-1-oak.zeng@intel.com> <20240117221223.18540-22-oak.zeng@intel.com> <2928ce36-06a6-4bee-b115-8dd61cc41dca@amd.com> From: =?UTF-8?Q?Christian_K=C3=B6nig?= In-Reply-To: X-ClientProxiedBy: FR3P281CA0003.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1d::22) To PH7PR12MB5685.namprd12.prod.outlook.com (2603:10b6:510:13c::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH7PR12MB5685:EE_|CH3PR12MB8726:EE_ X-MS-Office365-Filtering-Correlation-Id: 42629766-3956-4580-bd9b-08dc390a9b82 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7/dGISAVxyxneHPI3y+NmKff/NzWUdkkueunCWRivQlT6GLcW2PHXEwafilJfKW95NZyekjWQ8pt1zUtQ6mfBIFCswbp8ka6UWgr5BSm4/lYZ+fIfDld7Fjlq/1AFwi/2S9+GZIE8kTWsV0erxaGWZeTdgyz5U49WPVJW8dtvAH84U8uoLLO1wfB7UWDwCfhKtPB3gpNOUW+n09MrrJGPPGmtotAmNxOLIWRx3SQXzU92Ch1dsRj5rqrZifylBbXVK0sEKEoQxE2ewuDXn8Apn2ey9SWC3IcbpOzJhlGcShjbNX/ZzBFDTFnfa7x8Ahj/iwDHRM16JZ2CU2Lux9+GWclYMYIsVvSO6Y45CAYbcoYuVqs9Hb2Uh/e4ciBTAZJeuN9DrtN/YIQ6u7iA4+YYnhFHmvvzTsK2JRGf+t1D+6HfE5gT46q90RU5aMqiwNwC6X9hjMPJE96HcCbEdKFjNH6P7oIXIymc1SP/t9jTRR5dKp2xrzhGHlxMC43PctNGUQN+ryBqvDrjTUvsoBZlD5awVAR1I7uiHjMvdBco8UJtcmEpo+Q8x39lIPyGSIFAZxLD2fDuyUSlJP47wVSFLTCIcmVwBlOdpC3CuXx9cwX+s10gY4pNGJ5Qb1C67W7 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR12MB5685.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Z1V1U1Axektla29XUWFIZ0l3RkJYRWNwdktncTQ1c05qbHJCZldGcFV0bFg1?= =?utf-8?B?S3Y3d09LZ0xadUc1YjJuTVhxcHhrRmV6NmxYbU9YZEgvQmR6c1lMVm1PRTUr?= =?utf-8?B?aFVaVEN5a1Fab29lQXg4NW9sdzRwcVlKbVVHZThMRE1UMStwVis2UG9hWmRT?= =?utf-8?B?dW9XTllyY0t4TmhOMzhvdjZibTdvT3lLQ21rRll3U1drSkdiMDdTMmF3MzVC?= =?utf-8?B?c1dKT0pBWTFVclJWUmZUa0RkeEhlKzUxMFE2em1VcmhBY2pqWGVTTFRKeElB?= =?utf-8?B?L1NkZE9KLzV1K05kbWxxcjQrb0JuOVpkdzJRMU9LTmFSR1A3OXdoakZobXlT?= =?utf-8?B?S0w4a3cySEkzSm5OeEZja3VyWURDUDVzdHA1a0FDcG4xZktuRlZrSkFLT0tY?= =?utf-8?B?QVRsWm1VOGE2UHhiV2JTbWlCU0Y3aGdZY0szYXJsRDYvQXQydzVmU3o5L2tI?= =?utf-8?B?WkZvM3pjVkRROEEyYUFEOVpDWHZ0czcyTnJCdGRMcVVzLzc1WUFxdGRRak5H?= =?utf-8?B?TGQ4cEpIQzZHM09WYktwcTgxMWZFQmlMdmU5am5wV1hINkVSU3B0TFNpVVFu?= =?utf-8?B?UjRKeDQwbElRQXorYVEyQ3IrN2UzNGFyaUhDQWRsTTkvcEljR1hZMTNDNXY5?= =?utf-8?B?WkVxYXg0czdNMjZlRXc5ZEZYMkJCeEJFNkQxUXQ0TFdBMFJ5YWtUR1pMNzFC?= =?utf-8?B?b0hSOHN2Q2U5WWZlQlVETzMzcmo0R0hCalBoeUpuWElBZ2NpK2ZVSzBTakxi?= =?utf-8?B?eEppSTU1dEtVcm0yNjhQS2x0VXlxR29BcCt4V1pHRTEyMGhDemlPZXpnWGV1?= =?utf-8?B?d0xwMXdnMERPOTA1alJZSFhIT2ZNaFFxYXhOSDFiZjdLdW1KSWM3Zm82OWdX?= =?utf-8?B?cllOY0xWMUYwbFUxZTBJV1VzK0g0bVA1TEtheTRjQmJCMnNtMEZZejREVlFr?= =?utf-8?B?cmdVOXl2YkRXWm5JcnUreVVIUWVIbUJEeXJzUm4yMi9KZVltSWowaW9TMVJq?= =?utf-8?B?aFFUbUtCc0psR1pxWE0rRVROVnkyWkFwMmpFaElQbDcvT0F1cWdVQTloTDY0?= =?utf-8?B?YTVYZ1BOYmEvNEl3U2k1Q0IvczZkeUpSS2ExTDV2ZTRpcENmL1dWR1lDdDQx?= =?utf-8?B?cW9qbnVVKzRYck1aZTlOcmRJbGtWNCswdDl5ckhGRkxDcmVxK3hwbzVvbVh3?= =?utf-8?B?bWRiQ2JIL0hPUWxydTVYR0NaZ05LdHEzUTNpOTFOYjlMK3RTWU1YcHljNE8w?= =?utf-8?B?WnMrVFR2RWFlejhFZXB6R291STVYdUExbnMrRjZOUTVLalRqK0tlUnI0TU1p?= =?utf-8?B?NDRQV0k3UTlNb1hzcUduLzcwQVFIVUNOYVAzU1c1ZktHMVZ1SzhWWnMxZjZn?= =?utf-8?B?eFJQeTBaMWNNMk54K3puSEFsZVVsYnRVTUQ3cyswWUZnS1kwSHk1WFo0RXJo?= =?utf-8?B?UGtrekhSdEs0dUZZYmlCakpVQ1NpbzN3V25UdTRrbzdXemxHSGZvMlJtbTY5?= =?utf-8?B?ZGd6eVV1LzFmeEFUSDBrRVNPVzVjSyttV1NsLzg1cFZQWnlvZ1VmU0lFRDBp?= =?utf-8?B?UzczVDNWdk5mY0RQczNPczZvZ1ZLS3cxQjRMM2M3cW04aHZHcnZwVitKZC9U?= =?utf-8?B?R2d3b3FMNVdPS25wekdvdy9Idm4zenV1Q1FsTnNDRFVEb1NIVmhPRVN5VWh0?= =?utf-8?B?WXBJTFplemthNitNY29qQ1A5UW5UcjBKalFORU9va0VQeElMNG5xZzdoVC9C?= =?utf-8?B?TVpkcEVWbkhrQWJzU1RHMUNBV3pyd0FkaWRENzZvbURXSFJRZTRPY1NoRHdH?= =?utf-8?B?NWZNWTltTEVpRllCQkJGTHVSOS9UZWZDUjJlWHdEeUl6MW9mUXVQdUtHR0Vi?= =?utf-8?B?ZlNlcURhQTFuMCs4QUt4dmR4TTBSVG5KQW9SaFNmcFhBTmtBd3l6YkNtS0dT?= =?utf-8?B?ZFJPS3M3YmdCYmRJSWVpRDBNamM5M1k2Ky9lQ05qVkFsbGhSbVVsNnJuVVM0?= =?utf-8?B?d2huTW56RE1EVXNIYS9IZHBUM2V5SnBXcUpZL2lOZVBDSHBOak1saDVBdHlU?= =?utf-8?B?MFVWT0wyWHdJejQyNGxyS25tR1o1OFE4cVY4VFRDS1RYZmR5UTY2eDNXdWVm?= =?utf-8?Q?CpSsn9MLb5pIDD7FlZ6Oniz4g?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 42629766-3956-4580-bd9b-08dc390a9b82 X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB5685.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Feb 2024 09:41:30.3743 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Mho86dCauwA6dA6jAHTfdwnGaJol2GpeHUp6hBdxqX4iiEQyhXunSB/uCjNNs9A1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8726 X-BeenThere: intel-xe@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Xe graphics driver List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-xe-bounces@lists.freedesktop.org Sender: "Intel-xe" --------------6thb4d74p0ITt3oJxb4h3ERf Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Am 28.02.24 um 20:51 schrieb Zeng, Oak: > > The mail wasn’t indent/preface correctly. Manually format it. > > *From:*Christian König > *Sent:* Tuesday, February 27, 2024 1:54 AM > *To:* Zeng, Oak ; Danilo Krummrich > ; Dave Airlie ; Daniel Vetter > ; Felix Kuehling ; > jglisse@redhat.com > *Cc:* Welty, Brian ; > dri-devel@lists.freedesktop.org; intel-xe@lists.freedesktop.org; > Bommu, Krishnaiah ; Ghimiray, Himal Prasad > ; Thomas.Hellstrom@linux.intel.com; > Vishwanathapura, Niranjana ; > Brost, Matthew ; Gupta, saurabhg > > *Subject:* Re: Making drm_gpuvm work across gpu devices > > Hi Oak, > > Am 23.02.24 um 21:12 schrieb Zeng, Oak: > > Hi Christian, > > I go back this old email to ask a question. > > > sorry totally missed that one. > > Quote from your email: > > “Those ranges can then be used to implement the SVM feature > required for higher level APIs and not something you need at the > UAPI or even inside the low level kernel memory management.” > > “SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This > should not have any influence on the design of the kernel UAPI.” > > There are two category of SVM: > > 1.driver svm allocator: this is implemented in user space,  i.g., > cudaMallocManaged (cuda) or zeMemAllocShared (L0) or > clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd > and our umd implemented clSVMAlloc and zeMemAllocShared on top of > gem_create/vm_bind. Range A..B of the process address space is > mapped into a range C..D of the GPU address space, exactly as you > said. > > 2.system svm allocator:  This doesn’t introduce extra driver API > for memory allocation. Any valid CPU virtual address can be used > directly transparently in a GPU program without any extra driver > API call. Quote from kernel Documentation/vm/hmm.hst: “Any > application memory region (private anonymous, shared memory, or > regular file backed memory) can be used by a device transparently” > and “to share the address space by duplicating the CPU page table > in the device page table so the same address points to the same > physical memory for any valid main memory address in the process > address space”. In system svm allocator, we don’t need that A..B > C..D mapping. > > It looks like you were talking of 1). Were you? > > > No, even when you fully mirror the whole address space from a process > into the GPU you still need to enable this somehow with an IOCTL. > > And while enabling this you absolutely should specify to which part of > the address space this mirroring applies and where it maps to. > > */[Zeng, Oak] /* > > Lets say we have a hardware platform where both CPU and GPU support > 57bit(use it for example. The statement apply to any address range) > virtual address range, how do you decide “which part of the address > space this mirroring applies”? You have to mirror the whole address > space [0~2^57-1], do you? As you designed it, the gigantic > vm_bind/mirroring happens at the process initialization time, and at > that time, you don’t know which part of the address space will be used > for gpu program. Remember for system allocator, *any* valid CPU > address can be used for GPU program.  If you add an offset to > [0~2^57-1], you get an address out of 57bit address range. Is this a > valid concern? > Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that. The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one. > > > I see the system svm allocator as just a special case of the driver > allocator where not fully backed buffer objects are allocated, but > rather sparse one which are filled and migrated on demand. > > */[Zeng, Oak] /* > > Above statement is true to me. We don’t have BO for system svm > allocator. It is a sparse one as we can sparsely map vma to GPU. Our > migration policy decide which pages/how much of the vma is > migrated/mapped to GPU page table. > > *//* > > The difference b/t your mind and mine is, you want a gigantic vma > (created during the gigantic vm_bind) to be sparsely populated to gpu. > While I thought vma (xe_vma in xekmd codes) is a place to save memory > attributes (such as caching, user preferred placement etc). All those > memory attributes are range based, i.e., user can specify range1 is > cached while range2 is uncached. So I don’t see how you can manage it > with the gigantic vma. Do you split your gigantic vma later to save > range based memory attributes? > Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code. So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well. So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different. Regards, Christian. > Regards, > > Oak > > > > Regards, > Christian. > --------------6thb4d74p0ITt3oJxb4h3ERf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit Am 28.02.24 um 20:51 schrieb Zeng, Oak:

 

The mail wasn’t indent/preface correctly. Manually format it.

 

 

From: Christian König <christian.koenig@amd.com>
Sent: Tuesday, February 27, 2024 1:54 AM
To: Zeng, Oak <oak.zeng@intel.com>; Danilo Krummrich <dakr@redhat.com>; Dave Airlie <airlied@redhat.com>; Daniel Vetter <daniel@ffwll.ch>; Felix Kuehling <felix.kuehling@amd.com>; jglisse@redhat.com
Cc: Welty, Brian <brian.welty@intel.com>; dri-devel@lists.freedesktop.org; intel-xe@lists.freedesktop.org; Bommu, Krishnaiah <krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>; Thomas.Hellstrom@linux.intel.com; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com>; Brost, Matthew <matthew.brost@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com>
Subject: Re: Making drm_gpuvm work across gpu devices

 

Hi Oak,

Am 23.02.24 um 21:12 schrieb Zeng, Oak:

Hi Christian,

 

I go back this old email to ask a question.


sorry totally missed that one.

 

Quote from your email:

“Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.”

“SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.”

 

There are two category of SVM:

1.       driver svm allocator: this is implemented in user space,  i.g., cudaMallocManaged (cuda) or zeMemAllocShared (L0) or clSVMAlloc(openCL). Intel already have gem_create/vm_bind in xekmd and our umd implemented clSVMAlloc and zeMemAllocShared on top of gem_create/vm_bind. Range A..B of the process address space is mapped into a range C..D of the GPU address space, exactly as you said.

2.       system svm allocator:  This doesn’t introduce extra driver API for memory allocation. Any valid CPU virtual address can be used directly transparently in a GPU program without any extra driver API call. Quote from kernel Documentation/vm/hmm.hst: “Any application memory region (private anonymous, shared memory, or regular file backed memory) can be used by a device transparently” and “to share the address space by duplicating the CPU page table in the device page table so the same address points to the same physical memory for any valid main memory address in the process address space”. In system svm allocator, we don’t need that A..B C..D mapping.

 

It looks like you were talking of 1). Were you?


No, even when you fully mirror the whole address space from a process into the GPU you still need to enable this somehow with an IOCTL.

And while enabling this you absolutely should specify to which part of the address space this mirroring applies and where it maps to.

 

 

[Zeng, Oak]

Lets say we have a hardware platform where both CPU and GPU support 57bit(use it for example. The statement apply to any address range) virtual address range, how do you decide “which part of the address space this mirroring applies”? You have to mirror the whole address space [0~2^57-1], do you? As you designed it, the gigantic vm_bind/mirroring happens at the process initialization time, and at that time, you don’t know which part of the address space will be used for gpu program. Remember for system allocator, *any* valid CPU address can be used for GPU program.  If you add an offset to [0~2^57-1], you get an address out of 57bit address range. Is this a valid concern?


Well you can perfectly mirror on demand. You just need something similar to userfaultfd() for the GPU. This way you don't need to mirror the full address space, but can rather work with large chunks created on demand, let's say 1GiB or something like that.

The virtual address space is basically just a hardware functionality to route memory accesses. While the mirroring approach is a very common use case for data-centers and high performance computing there are quite a number of different use cases which makes use of virtual address space in a non "standard" fashion. The native context approach for VMs is just one example, databases and emulators are another one.



I see the system svm allocator as just a special case of the driver allocator where not fully backed buffer objects are allocated, but rather sparse one which are filled and migrated on demand.

 

 

[Zeng, Oak]

Above statement is true to me. We don’t have BO for system svm allocator. It is a sparse one as we can sparsely map vma to GPU. Our migration policy decide which pages/how much of the vma is migrated/mapped to GPU page table.

 

The difference b/t your mind and mine is, you want a gigantic vma (created during the gigantic vm_bind) to be sparsely populated to gpu. While I thought vma (xe_vma in xekmd codes) is a place to save memory attributes (such as caching, user preferred placement etc). All those memory attributes are range based, i.e., user can specify range1 is cached while range2 is uncached. So I don’t see how you can manage it with the gigantic vma. Do you split your gigantic vma later to save range based memory attributes?


Yes, exactly that. I mean the splitting and eventually merging of ranges is a standard functionality of the GPUVM code.

So when you need to store additional attributes per range then I would strongly suggest to make use of this splitting and merging functionality as well.

So basically an IOCTL which says range A..B of the GPU address space is mapped to offset X of the CPU address space with parameters Y (caching, migration behavior etc..). That is essentially the same we have for mapping GEM objects, the provider of the backing store is just something different.

Regards,
Christian.

 

Regards,

Oak



Regards,
Christian.

 


--------------6thb4d74p0ITt3oJxb4h3ERf--