From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32D1FC48BDF for ; Sun, 20 Jun 2021 14:15:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A38046109E for ; Sun, 20 Jun 2021 14:15:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A38046109E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=mit.edu Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC7576B006E; Sun, 20 Jun 2021 10:14:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B77926B0070; Sun, 20 Jun 2021 10:14:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A185A6B0072; Sun, 20 Jun 2021 10:14:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 6B78E6B006E for ; Sun, 20 Jun 2021 10:14:59 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EB672181AEF0B for ; Sun, 20 Jun 2021 14:14:58 +0000 (UTC) X-FDA: 78274298676.11.B485AF8 Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by imf22.hostedemail.com (Postfix) with ESMTP id 62A18C0237E4 for ; Sun, 20 Jun 2021 14:14:55 +0000 (UTC) Received: from cwcc.thunk.org (pool-72-74-133-215.bstnma.fios.verizon.net [72.74.133.215]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 15KEEsxi001510 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 20 Jun 2021 10:14:55 -0400 Received: by cwcc.thunk.org (Postfix, from userid 15806) id 4EFFB15C3C9F; Sun, 20 Jun 2021 10:14:54 -0400 (EDT) Date: Sun, 20 Jun 2021 10:14:54 -0400 From: "Theodore Ts'o" To: Alex Sierra Cc: akpm@linux-foundation.org, Felix.Kuehling@amd.com, linux-mm@kvack.org, rcampbell@nvidia.com, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, hch@lst.de, jgg@nvidia.com, jglisse@redhat.com Subject: Re: [PATCH v3 0/8] Support DEVICE_GENERIC memory in migrate_vma_* Message-ID: References: <20210617151705.15367-1-alex.sierra@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210617151705.15367-1-alex.sierra@amd.com> Authentication-Results: imf22.hostedemail.com; dkim=none; spf=none (imf22.hostedemail.com: domain of tytso@mit.edu has no SPF policy when checking 18.9.28.11) smtp.mailfrom=tytso@mit.edu; dmarc=none X-Stat-Signature: uqw9teo6ymtdw51atiwzhfm1w6eiop5p X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 62A18C0237E4 X-HE-Tag: 1624198495-478689 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 17, 2021 at 10:16:57AM -0500, Alex Sierra wrote: > v1: > AMD is building a system architecture for the Frontier supercomputer with a > coherent interconnect between CPUs and GPUs. This hardware architecture allows > the CPUs to coherently access GPU device memory. We have hardware in our labs > and we are working with our partner HPE on the BIOS, firmware and software > for delivery to the DOE. > > The system BIOS advertises the GPU device memory (aka VRAM) as SPM > (special purpose memory) in the UEFI system address map. The amdgpu driver looks > it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC > using devm_memremap_pages. > > Now we're trying to migrate data to and from that memory using the migrate_vma_* > helpers so we can support page-based migration in our unified memory allocations, > while also supporting CPU access to those pages. > > This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave > correctly in the migrate_vma_* helpers. We are looking for feedback about this > approach. If we're close, what's needed to make our patches acceptable upstream? > If we're not close, any suggestions how else to achieve what we are trying to do > (i.e. page migration and coherent CPU access to VRAM)? Is there a way we can test the codepaths touched by this patchset? It doesn't have to be via a complete qemu simulation of the GPU device memory, but some way of creating MEMORY_DEVICE_GENERIC subject to migrate_vma_* helpers so we can test for regressions moving forward. Thanks, - Ted