From: Andrew Morton <akpm@linux-foundation.org>
To: Alex Sierra <alex.sierra@amd.com>
Cc: <Felix.Kuehling@amd.com>, <linux-mm@kvack.org>,
<rcampbell@nvidia.com>, <linux-ext4@vger.kernel.org>,
<linux-xfs@vger.kernel.org>, <amd-gfx@lists.freedesktop.org>,
<dri-devel@lists.freedesktop.org>, <hch@lst.de>, <jgg@nvidia.com>,
<jglisse@redhat.com>, <apopple@nvidia.com>
Subject: Re: [PATCH v1 00/12] MEMORY_DEVICE_COHERENT for CPU-accessible coherent device memory
Date: Tue, 12 Oct 2021 11:39:57 -0700 [thread overview]
Message-ID: <20211012113957.53f05928dd60f3686331fede@linux-foundation.org> (raw)
In-Reply-To: <20211012171247.2861-1-alex.sierra@amd.com>
On Tue, 12 Oct 2021 12:12:35 -0500 Alex Sierra <alex.sierra@amd.com> wrote:
> This patch series introduces MEMORY_DEVICE_COHERENT, a type of memory
> owned by a device that can be mapped into CPU page tables like
> MEMORY_DEVICE_GENERIC and can also be migrated like MEMORY_DEVICE_PRIVATE.
> With MEMORY_DEVICE_COHERENT, we isolate the new memory type from other
> subsystems as far as possible, though there are some small changes to
> other subsystems such as filesystem DAX, to handle the new memory type
> appropriately.
>
> We use ZONE_DEVICE for this instead of NUMA so that the amdgpu
> allocator can manage it without conflicting with core mm for non-unified
> memory use cases.
>
> How it works: The system BIOS advertises the GPU device memory (aka VRAM)
> as SPM (special purpose memory) in the UEFI system address map.
> The amdgpu driver registers the memory with devmap as
> MEMORY_DEVICE_COHERENT using devm_memremap_pages.
>
> The initial user for this hardware page migration capability will be
> the Frontier supercomputer project.
To what other uses will this infrastructure be put?
Because I must ask: if this feature is for one single computer which
presumably has a custom kernel, why add it to mainline Linux?
> Our nodes in the lab have .5 TB of
> system memory plus 256 GB of device memory split across 4 GPUs, all in
> the same coherent address space. Page migration is expected to improve
> application efficiency significantly. We will report empirical results
> as they become available.
>
> This includes patches originally by Ralph Campbell to change ZONE_DEVICE
> reference counting as requested in previous reviews of this patch series
> (see https://patchwork.freedesktop.org/series/90706/). We extended
> hmm_test to cover migration of MEMORY_DEVICE_COHERENT. This patch set
> builds on HMM and our SVM memory manager already merged in 5.14.
> We would like to complete review and merge this migration patchset for
> 5.16.
next prev parent reply other threads:[~2021-10-12 18:40 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-12 17:12 [PATCH v1 00/12] MEMORY_DEVICE_COHERENT for CPU-accessible coherent device memory Alex Sierra
2021-10-12 17:12 ` [PATCH v1 01/12] ext4/xfs: add page refcount helper Alex Sierra
2021-10-12 17:12 ` [PATCH v1 02/12] mm: remove extra ZONE_DEVICE struct page refcount Alex Sierra
2021-10-12 17:12 ` [PATCH v1 03/12] mm: add zone device coherent type memory support Alex Sierra
2021-10-12 17:12 ` [PATCH v1 04/12] mm: add device coherent vma selection for memory migration Alex Sierra
2021-10-12 17:12 ` [PATCH v1 05/12] drm/amdkfd: ref count init for device pages Alex Sierra
2021-10-12 17:12 ` [PATCH v1 06/12] drm/amdkfd: add SPM support for SVM Alex Sierra
2021-10-12 17:12 ` [PATCH v1 07/12] drm/amdkfd: coherent type as sys mem on migration to ram Alex Sierra
2021-10-12 17:12 ` [PATCH v1 08/12] lib: test_hmm add ioctl to get zone device type Alex Sierra
2021-10-12 17:12 ` [PATCH v1 09/12] lib: test_hmm add module param for " Alex Sierra
2021-10-12 17:12 ` [PATCH v1 10/12] lib: add support for device coherent type in test_hmm Alex Sierra
2021-10-12 17:12 ` [PATCH v1 11/12] tools: update hmm-test to support device coherent type Alex Sierra
2021-10-12 17:12 ` [PATCH v1 12/12] tools: update test_hmm script to support SP config Alex Sierra
2021-10-12 18:39 ` Andrew Morton [this message]
2021-10-12 18:56 ` [PATCH v1 00/12] MEMORY_DEVICE_COHERENT for CPU-accessible coherent device memory Jason Gunthorpe
2021-10-12 19:03 ` Andrew Morton
2021-10-12 23:04 ` Felix Kuehling
2021-10-13 13:34 ` Daniel Vetter
2021-10-12 19:00 ` Felix Kuehling
2021-10-12 19:11 ` Matthew Wilcox
2021-10-12 20:24 ` Felix Kuehling
2021-10-12 20:44 ` Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211012113957.53f05928dd60f3686331fede@linux-foundation.org \
--to=akpm@linux-foundation.org \
--cc=Felix.Kuehling@amd.com \
--cc=alex.sierra@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=apopple@nvidia.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=hch@lst.de \
--cc=jgg@nvidia.com \
--cc=jglisse@redhat.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-xfs@vger.kernel.org \
--cc=rcampbell@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).