[PATCH v17 00/12] migrate on fault for device pages

[PATCH v17 00/12] migrate on fault for device pages

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Currently, the way device page faulting and migration works
is not optimal, if you want to do both fault handling and
migration at once.

Being able to migrate not present pages (or pages mapped with incorrect
permissions, eg. COW) to the GPU requires doing either of the
following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
   a) call hmm_range_fault() to fault pages in
   b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two
page walks even when most of the time the pages are present or zero page
mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if
pages aren't present because now you have to walk the page tables three
times (once to find the page is not present, once so hmm_range_fault()
can find a non-present page to fault in and once again to setup the
migration). It is also tricky to code correctly. One page table walk
could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting
pages in as required and replacing them with migration entries if
requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling.
For the migrate_vma_setup() call paths, new flags, MIGRATE_VMA_FAULT,
and MIGRATE_VMA_WRITE are added to tell to add fault handling to migrate.

An extra benefit of migrating with hmm_range_fault() path
is the migrate_vma.vma gets populated, so no need to
retrieve that separataly.

This series avoids the problems in current collecting implementation,
where racing MADV_DONTNEED could make the collecting arrays overflow.
This is achieved mainly by having a rollback mechanism and
populating the collecting array as the last step.

Tested in X86-64 VM with HMM test device, passing the selftests.
The main goal of this patchset is to provide a new feature,
migrate efficiently on fault, and not directly improve performance
of current migrate_vma_setup() (which is now routed thru
hmm_range_fault()) callers. That said, the migrate throughput tests
from the selftests show that there is at least no regression for
current migrate_vma_setup() callers.

For nouveau migrate-on-fault test harness was : Fedora 44,
Mesa Rusticl, and a fine-grained-system-SVM OpenCL program.
Nouveau SVM is opt-in via Mesa. The OpenCL program passes
a plain malloc pointer (64 pages, array of integers) to the GPU.
Increments each element on GPU, and verifies on the CPU the result.
So it drives both directions of migration. With svm_migrate_on_fault=1,
no migration at bind happens. The GPU used in testing was Pascal
generation GP102.

Changes in v17:
  - rebased on v7.4-rc1 (drm-tip)
  - addressed sashiko's concerns
  - addressed review comments from Balbir
  - fixed email address of Liam R. Howlett

Sashiko's concerns in v16
https://sashiko.dev/#/patchset/20261021060506.2377545-1-mpenttil%40redhat.com

Answers to sashiko's concerns per patch:
patch 03
  Q "Can this condition in mm/hmm.c:hmm_vma_capture_migrate_range() ever
  be true?"
  A it can't trigger; dropped in v17
  Q "Should this check instead verify that the single VMA covers the
  entire requested range..?"
  A In v17 the single vma must cover the whole range (-ERANGE otherwise,
  like migrate_vma_setup()), and hmm_vma_walk_hole() returns -EFAULT for
  holes outside a vma when migrating. That handles both the leading hole
  and the whole-range hole."  
  Q "is it intentional that hmm_vma_capture_migrate_range() runs before
  validating vma->vm_flags?"
  A  Not intentional. In v17 the capture runs only after the vm_flags check,
  so no MMU_NOTIFY_MIGRATE invalidation is fired for a vma we can't handle,
  and an unsupported vma fails with -EFAULT when migrating. I also added
  the same hugetlb/VM_SPECIAL/DAX restrictions as migrate_vma_setup().
  Q "Should VMA eligibility for both HMM and migration be verified before
  capturing the VMA and arming the MMU notifier?"
  A Fixed in v17
  Q.1 "Invalidation range across faults question"
  A.2 In v17 it's closed before faulting and reopened for the next walk, so
  it only covers walking and collecting
  Q.2 "Nested invalidations make the retry check return -EBUSY"
  A.2 This is pre existig behavior already in hmm_range_walk(). -EBUSY and
  retries is normal HMM behavior and with migration the caller finalizes the
  collected so far entries before retry
  Q.3 "Re-collecting a page without a fresh invalidate_range_start()..?"
  A.3 One start/end pair covers every PTE change in the range until the
  end call

patch 04
  Q "about skipping migration entries pte and pmd level"
  A In v17 both paths are fixed: when migrating, a foreign migration entry
  with a fault requested now goes through hmm_do_fault() and waits outside
  the walk, otherwise the pfns are masked as before; our own collected
  entries are recognised and kept. The input pfns are also cleared when
  migrating, so nothing is left uninitialized.
  Q "stale enries after EAGAIN->goto again?"
  A The !ptep rollback did target the wrong range. In v17, after the
  pte lock is dropped (split or table gone) the walk restarts the pmd
  range from start: our own migration entries are recognised
  (collected and same page) and kept, everything else collected is rolled
  back as it's revisited. The pmd migration entry path now rolls back
  before masking, and the own-entry check there requires the same page
  Q "Behavior after failed splits"
  A Fixed that after fail, return what we have so for, no HMM_PFN_ER

patch 05
  Q1 "About folio splits.."
  A But made a fix for v17 so that check if folio is large before split
  so no error if already split
  Q2 Can this trigger a use-after-free when fault_page is a tail subpage
  of a large device-private folio?
  A A fix is in v17
  Q3 "about protnone case for migrate_vma_setup() callers without
  MIGRATE_VMA_FAULT"
  Q4 "Doesn't that make the !pte_present(pte) device-private migration
  path here unreachable unless range->dev_private_owner is also set to
  migrate->pgmap_owner?"
  A yes both fixed in v17
  Q5 "Zero-pfn branch wipes INOUT flags, no vma_is_anonymous() check"
  A fix is in v17
  Q6 "When migrate_vma_split_folio() fails (both here and in the
  !pte_present(pte) branch above), hmm_vma_handle_migrate_prepare() jumps
  to out_error and returns -EFAULT"
  A This is the same as the earlier split-failure comment: in v17 a
  failed split no longer reports HMM_PFN_ERROR, and addr keeps its valid
  pfn (pmd level too). Skipping the rest of the pmd range is intentional
  and matches existing behavior.

  Q6.1: "Could folio_lock held before handle_mm_fault() cause deadlock"
  A No. Collection is anonymous-only: file folios are never collected, so
  page cache and invalidate_lock ordering (truncate -> folio_lock) never
  come into play. Faults are confined to the single vma being migrated,
  and a non-KSM anonymous page is mapped at most once in it ; that mapping
  is our migration entry. A PMD-mapped THP is collected whole, behind its
  PMD migration entry; PTE-mapped large folios are split before collecting.

  We never fault our own entries: when migrating, only the PTE
  that needs a fault is faulted, so migration_entry_wait() on our own
  lock can't happen.

  KSM (the same page at several addresses): the second occurrence fails
  the collection trylock and isn't collected, and breaking COW
  (do_wp_page()) copies without taking the lock.

  A swapcache folio still referenced by other swap entries isn't
  collected (folio_free_swap() check); otherwise do_swap_page(), with
  no retry allowed while migrating, would block on our lock.

  Reclaim and compaction under the fault's allocations: reclaim only
  trylocks, and direct compaction runs with PF_MEMALLOC and never takes
  a blocking folio lock; it also skips our folios because of the
  extra reference. Other tasks reach our folios only through the
  migration entries (migration_entry_wait(), holding no folio locks)
  or via trylock.

  Q6.2 and Q6.3 are addressed in v17:  own migration entries are recognised
  before the required_fault check and kept on re-walk, and the !ptep case
  re-walks from start without the wrong-range rollback, rolling back
  collected entries when revisited

patch 06
  Q1 "Can returning -EBUSY from the split path in
  hmm_vma_handle_migrate_prepare_pmd() leave stale pre-split PMD entries in
  range->hmm_pfns[] across the re-walk?"
  A In v17 the re-walk always rewrites or faults these entries: a migration
  entry is only kept as ours if it's collected (VALID|MIGRATE) and the
  same page, so the stale pmd output never survives; otherwise it's faulted
  or reset via out:. Migration takes requests only from default_flags
  (pfn_flags_mask must be 0, input is cleared), so there are no per-page
  request flags to clobber. v17 clears the pmd-level entries before
  re-walking as PTEs.
  
  for Q2 : done in v17: both the pmd folio_trylock() failure and a
  failed pmd split now return 0, leaving the valid pfns from
  hmm_vma_handle_pmd() without HMM_PFN_MIGRATE, the same as the
  PTE-level trylock failure. The HMM_PFN_ERROR fill in
  hmm_vma_walk_pmd() is gone.

patch 08
  Q "What happens if entries in hmm_pfns were previously collected,
  locks were dropped, and on re-walk the page table entry now requires
  a fault, fails with -EFAULT, or is a PMD migration entry?"
  A these should roll back first. fixed in v17
  Q "Could reaching the out: label in hmm_vma_handle_pte() prematurely
  roll back a migration PTE that this walk already installed?"
  A Addressed in v17. Our own migration entries are recognised before the
  required_fault check (collected and the same page) and kept, so a
  re-walk never rolls them back under a live migration pte. A migration
  entry that isn't ours is rolled back (at out:, or before faulting) and,
  with REQ_FAULT while migrating, goes through hmm_do_fault() so
  handle_mm_fault() waits for that migration outside the walk.
  Q "Does this rollback miss entries collected in [start, addr)
  when the PTE lock is dropped partway through a PMD?"
  A The !ptep rollback over the wrong range fixed in v17

patch 09
   same concerns as for patch 03-05 and 08 and addressed in v17

patch 10
  Q "What happens in do_fault_and_migrate() if mmu_interval_read_retry()
  returns true after migrate_vma_pages() has already migrated the folios?"
  A The state is always consistent: any invalidation also clears the
  device page table, so there's never a stale device mapping.
  Normally the trip comes from an unmap/zap: our migration ptes are gone,
  migrate_vma_finalize() releases the dst pages, and the -EBUSY retry
  re-migrates. In the remaining case (e.g. a huge zero pmd split in
  migrate_vma_insert_page()) the pages end up committed with no device
  mapping yet, which is legal and filled in by the next device fault
  with real HW. The selftest never hits that case, since the buffer
  is written first.
  Q "Does TEST_F(hmm, migrate_on_fault) actually trigger page faults
  during hmm_range_fault()?"
  A  Agreed, with the whole buffer written it never faulted. In v17
  the test uses hmm_buffer_alloc() and, like anon_read, leaves the
  first two pages unwritten: page 0 stays pte_none so hmm_range_fault()
  has to fault it in before migrating, and page 1 maps the zero page.
  Both are checked to read back as zero from the device.

patch 11
  sashiko flags some previously flagged issues in the documentation
  patch

patch 12
  Q What happens in nouveau_dmem_fence_poll() in
  drivers/gpu/drm/nouveau/nouveau_dmem.c when nouveau_fence_new()
  fails in its callers?
  A Agreed, the return value was ignored, as in the existing
  nouveau_dmem code. In v17 both paths that use the poll initialize
  the fence to NULL, check nouveau_fence_new(), and treat a failure
  as an unconfirmed copy: readback returns VM_FAULT_SIGBUS, and the
  fault path doesn't map the page. The same pattern in the existing
  bulk migration path can be fixed separately.
  
  Q "When the retry loop in nouveau_dmem_migrate_fault() in
  drivers/gpu/drm/nouveau/nouveau_dmem.c repeats after
  ret == -EBUSY (either from hmm_range_fault() or from
  mmu_interval_read_retry()), does migrate.src (aliased by
  range.hmm_pfns) retain stale migrate_pfn values from the
  previous iteration?"
  A In v17 hmm_range_fault() clears range->hmm_pfns[] at the
  start of every migrating walk (requests come only from default_flags),
  so each retry starts from zero, the same as migrate_vma_setup()
  zeroing args->src, and there's no early-return path left
  that keeps an input entry. migrate_hmm_range_setup() also
  resets dst[] for every entry.
  
  Q Question about drm->dmem->migrate.chan synchronization
  A Copies from migrate_to_ram(), the SVM fault worker, bulk
  migration and eviction could write commands and fences into
  the shared migrate channel concurrently.
  In v17 those writes are serialized by a dmem migrate mutex,
  held only around the command writes and fence creation,
  not across page allocation. Serializing against TTM's own use
  of the channel is outside this series.
  
  Q "If nouveau_dmem_fence_poll(&fence) times out and returns
  -EBUSY here in nouveau_dmem_migrate_fault()
  (or above in nouveau_dmem_migrate_to_ram()) in
  drivers/gpu/drm/nouveau/nouveau_dmem.c, doesn't the page
  migration still get committed with uncopied data while DMA
  may still be active?"
  A In v17 the migration is committed only after the copy
  fence completes: migrate_vma_pages() now runs after the
  wait (the fault path still replays before waiting). If the
  copy can't be confirmed (timeout or no fence), the entries
  are marked not migrated, so finalize restores the original
  ptes and the source stays valid; the GPU fallback then maps
  the original page. Since a wedged copy engine may still use
  the pages, their DMA mappings and the destination folios
  are leaked on purpose rather than unmapped and freed.
  
  Q "Can nouveau_dmem_migrate_fault() in
  drivers/gpu/drm/nouveau/nouveau_dmem.c
  map a VRAM page into the GPU that migrate_vma_pages() rejected
  and that migrate_vma_finalize() is about to free?"
  A  In v17 the fault path clears the GPU pfn for every entry
  that migrate_vma_pages() didn't migrate
  (MIGRATE_PFN_MIGRATE cleared), so a rejected destination page
  is never mapped. The existing fallback then maps the original
  source page in place, which finalize restores.
  
  Q "Why is __nouveau_pfns_map_locked() called before checking
  whether pfns[0] & NVIF_VMM_PFNMAP_V0_V is set in
  nouveau_dmem_migrate_fault() in
  drivers/gpu/drm/nouveau/nouveau_dmem.c?"
  A In v17 the GPU is only mapped when the resulting pfn is valid.
  If nothing migrated and the snapshot isn't mappable
  (huge zero folio, error entry, failed VRAM allocation), leave
  the existing GPU mapping untouched and return -ENOENT, so the
  caller falls back to nouveau_range_fault().
  
  Q "Doesn't ignoring all MMU_NOTIFY_MIGRATE events with
  range->owner == sn->svmm->vmm->cli->drm->dev in
  nouveau_svm_range_invalidate() in
  drivers/gpu/drm/nouveau/nouveau_svm.c also ignore concurrent
  migrations triggered by other paths on the same device, allowing
  freed VRAM or system RAM pages to be mapped into the GPU VMM?"  
  A In v17 only the invalidations raised by this fault's own
  migration are ignored: the per-fault notifier records the
  task running the walk, and only MIGRATE events from that
  task while it migrates skip the seq bump.

  Q "Should __nouveau_pfns_map_locked() in
  drivers/gpu/drm/nouveau/nouveau_svm.c propagate the return value
  of nvif_object_ioctl() to nouveau_dmem_migrate_fault()?"
  A  Yes. In v17 __nouveau_pfns_map_locked() returns the
  nvif_object_ioctl() result and nouveau_dmem_migrate_fault()
  propagates it
  
Link to v16:
https://lore.kernel.org/linux-mm/20261021060506.2377545-1-mpenttil@redhat.com/

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Lyude Paul <lyude@redhat.com>
Cc: Danilo Krummrich <dakr@kernel.org>

Mika Penttilä (12):
  mm/Kconfig: changes for migrate on fault for device pages
  mm: add helper to convert HMM pfn to migrate pfn
  mm/hmm: preparations for HMM to participate in migration
  mm/hmm: do the plumbing for HMM to participate in migration
  mm/hmm: migrate collection in HMM pagewalk - pte level
  mm/hmm: migrate collection in HMM pagewalk - pmd level
  mm/hmm: add lazy MMU mode support for migration in HMM pagewalk
  mm/hmm: implement rollback for device page migration in HMM pagewalk
  mm: enable device page migration from HMM pagewalk
  lib/test_hmm: add a new testcase for the migrate on fault
  Documentation/mm/hmm: document migration through hmm_range_fault()
  drm/nouveau/svm: add support for migrate on fault

 Documentation/mm/hmm.rst               |   45 +
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  402 ++++++++-
 drivers/gpu/drm/nouveau/nouveau_dmem.h |   12 +
 drivers/gpu/drm/nouveau/nouveau_svm.c  |  155 ++--
 drivers/gpu/drm/nouveau/nouveau_svm.h  |    5 +
 include/linux/hmm.h                    |   51 +-
 include/linux/migrate.h                |   59 +-
 lib/test_hmm.c                         |  174 +++-
 lib/test_hmm_uapi.h                    |   21 +-
 mm/Kconfig                             |    1 +
 mm/hmm.c                               | 1075 ++++++++++++++++++++++--
 mm/migrate_device.c                    |  618 +++-----------
 tools/testing/selftests/mm/hmm-tests.c |   56 ++
 13 files changed, 1984 insertions(+), 690 deletions(-)

drm-tip
base-commit: abc364961b361ab626fa5a09568b22744f1e0393
-- 
2.55.0

[PATCH v17 01/12] mm/Kconfig: changes for migrate on fault for device pages

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

HMM depends on MMU notifiers. With the unified HMM/migrate_device
page table walk migrate_device needs HMM enabled.
Enable them explicitly to avoid breaking random configs.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Balbir Singh <balbirs@nvidia.com>
---
 mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..6fa03655b575 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -682,6 +682,7 @@ config MIGRATION
 
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
+	select HMM_MIRROR
 
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
-- 
2.55.0

[PATCH v17 02/12] mm: add helper to convert HMM pfn to migrate pfn

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

The unified HMM/migrate_device pagewalk does the "collecting"
on the HMM side, so we need a helper to transfer pfns to the
migrate_vma world.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 include/linux/hmm.h     | 29 +++++++++++++++++-----
 include/linux/migrate.h |  3 ++-
 mm/migrate_device.c     | 55 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 80 insertions(+), 7 deletions(-)

diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 6f04e3932f5b..4f56f3419cb4 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -13,6 +13,8 @@
 
 struct mmu_interval_notifier;
 
+struct migrate_vma;
+
 /*
  * On output:
  * 0             - The page is faultable and a future call with 
@@ -27,13 +29,22 @@ struct mmu_interval_notifier;
  * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
  * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
  *                      to mark that page is already DMA mapped
+ * HMM_PFN_MIGRATE    - The entry is to be migrated. Note, HMM_PFN_MIGRATE
+ *                      alone without HMM_PFN_VALID denotes the
+ *                      empty page.
+ *                      This flag optionally together with HMM_PFN_COMPOUND
+ *                      are indicators for migrate_hmm_range_setup() to
+ *                      setup the migrate pfns.
+ * HMM_PFN_COMPOUND   - The entry represents a > 0 order page
  *
  * On input:
- * 0                 - Return the current state of the page, do not fault it.
- * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_fault()
- *                     will fail
- * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
- *                     will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * 0                   - Return the current state of the page, do not fault it.
+ * HMM_PFN_REQ_FAULT   - The output must have HMM_PFN_VALID or hmm_range_fault()
+ *                       will fail
+ * HMM_PFN_REQ_WRITE   - The output must have HMM_PFN_WRITE or hmm_range_fault()
+ *                       will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * HMM_PFN_REQ_MIGRATE - For default_flags only, request to migrate the range,
+ *                       according to hmm_range.migrate.flags
  */
 enum hmm_pfn_flags {
 	/* Output fields and flags */
@@ -48,11 +59,15 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),
 
 	/* Input flags */
 	HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
 	HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,
+	HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
 
 	HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
 };
@@ -107,6 +122,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
  * @default_flags: default flags for the range (write, read, ... see hmm doc)
  * @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter
  * @dev_private_owner: owner of device private pages
+ * @migrate: structure for migrating a range of a VMA
  */
 struct hmm_range {
 	struct mmu_interval_notifier *notifier;
@@ -117,6 +133,7 @@ struct hmm_range {
 	unsigned long		default_flags;
 	unsigned long		pfn_flags_mask;
 	void			*dev_private_owner;
+	struct migrate_vma      *migrate;
 };
 
 /*
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 78424b3824c2..280ec9a866c7 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -3,6 +3,7 @@
 #define _LINUX_MIGRATE_H
 
 #include <linux/mm.h>
+#include <linux/hmm.h>
 #include <linux/mempolicy.h>
 #include <linux/migrate_mode.h>
 #include <linux/hugetlb.h>
@@ -202,7 +203,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
 			unsigned long *dst_pfns, unsigned long npages);
-
+void migrate_hmm_range_setup(struct hmm_range *range);
 #endif /* CONFIG_MIGRATION */
 
 #endif /* _LINUX_MIGRATE_H */
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 9e19fadfa65c..ba6775a0d37e 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -1530,3 +1530,58 @@ int migrate_device_coherent_folio(struct folio *folio)
 		return 0;
 	return -EBUSY;
 }
+
+/**
+ * migrate_hmm_range_setup() - prepare to migrate a range of memory
+ * @range: contains pointer to struct migrate_vma to be set up.
+ *
+ * When collecting has been done with hmm_range_fault(), this
+ * should be called next, and completes range->migrate by
+ * populating migrate->src[] and migrate->dst[]
+ * using range->hmm_pfns[].
+ * Also, migrate->cpages and migrate->npages get initialized.
+ * After migrate_hmm_range_setup(), range->migrate is good
+ * for the rest of the migrate_vma_* flow.
+ */
+void migrate_hmm_range_setup(struct hmm_range *range)
+{
+	struct migrate_vma *migrate = range->migrate;
+
+	if (!migrate)
+		return;
+
+	migrate->npages = (migrate->end - migrate->start) >> PAGE_SHIFT;
+	migrate->cpages = 0;
+
+	for (unsigned long i = 0; i < migrate->npages; i++) {
+		unsigned long pfn = range->hmm_pfns[i];
+
+		/*
+		 * We are only interested in entries to be
+		 * migrated.
+		 */
+		if (!(pfn & HMM_PFN_MIGRATE)) {
+			migrate->src[i] = 0;
+			migrate->dst[i] = 0;
+			continue;
+		}
+
+		migrate->cpages++;
+
+		/* HMM_PFN_MIGRATE without HMM_PFN_VALID denotes the special zero page */
+		if (pfn & HMM_PFN_VALID)
+			migrate->src[i] = migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn)));
+		else
+			migrate->src[i] = 0;
+
+		migrate->src[i] |= MIGRATE_PFN_MIGRATE;
+		migrate->src[i] |= (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0;
+		migrate->src[i] |= (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0;
+		migrate->dst[i] = 0;
+	}
+
+	if (migrate->cpages)
+		migrate_vma_unmap(migrate);
+
+}
+EXPORT_SYMBOL(migrate_hmm_range_setup);
-- 
2.55.0

[PATCH v17 03/12] mm/hmm: preparations for HMM to participate in migration

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

For migration to happen after hmm_range_fault(), the vma has to be
retrieved. Luckily, pagewalk already resolves that for us, so
just have to save it for further use. For migration, mmap_lock
has to be held for the whole operation, so vma stays stable.
In case lock-drop behavior is requested, return an error while
migrating. Also, rename some struct hmm_vma_walk fields to solve
namespace collisions, and we are later introducing more
xxlocked fields so be unambiguous.

Take care also of firing the mmu_notifier when migrating.

When migrating, requests come only from range->default_flags
(pfn_flags_mask must be 0), so the input range->hmm_pfns carry
nothing and are cleared at the start of hmm_range_fault(). Collected
entries (HMM_PFN_VALID | HMM_PFN_MIGRATE) can then be told apart
reliably during the walk and on rollback. This also means that
HMM_PFN_INOUT_FLAGS such as HMM_PFN_DMA_MAPPED are not preserved
across a migrating call, which is intended since the pages are being
replaced.

Also, prepare for differences of how mm_struct * is retrieved
in fault and migrate_vma paths.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 include/linux/migrate.h |  50 ++++++++++++-
 lib/test_hmm.c          |   2 +-
 mm/hmm.c                | 157 ++++++++++++++++++++++++++++++++++------
 3 files changed, 186 insertions(+), 23 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 280ec9a866c7..dd7dd7ec6995 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -5,6 +5,7 @@
 #include <linux/mm.h>
 #include <linux/hmm.h>
 #include <linux/mempolicy.h>
+#include <linux/mmu_notifier.h>
 #include <linux/migrate_mode.h>
 #include <linux/hugetlb.h>
 
@@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 	spin_unlock(ptl);
 }
 
+enum migrate_vma_info {
+	MIGRATE_VMA_SELECT_NONE = 0,
+	MIGRATE_VMA_SELECT_COMPOUND = MIGRATE_VMA_SELECT_NONE,
+};
+
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+	return MIGRATE_VMA_SELECT_NONE;
+}
+
+static inline void hmm_fill_migrate_vma(struct hmm_range *range,
+					struct vm_area_struct *vma,
+					unsigned long start,
+					unsigned long end)
+{
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
+{
+	return range->notifier->mm;
+}
+#endif
 #endif /* CONFIG_MIGRATION */
 
 #ifdef CONFIG_NUMA_BALANCING
@@ -151,7 +175,8 @@ static inline unsigned long migrate_pfn(unsigned long pfn)
 	return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID;
 }
 
-enum migrate_vma_direction {
+enum migrate_vma_info {
+	MIGRATE_VMA_SELECT_NONE = 0,
 	MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
 	MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
 	MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
@@ -193,6 +218,29 @@ struct migrate_vma {
 	struct page		*fault_page;
 };
 
+// TODO: enable migration
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+	return 0;
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
+{
+	return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm;
+}
+#endif
+
+static inline void hmm_fill_migrate_vma(struct hmm_range *range,
+					struct vm_area_struct *vma,
+					unsigned long start,
+					unsigned long end)
+{
+	range->migrate->vma   = vma;
+	range->migrate->start = start;
+	range->migrate->end   = end;
+}
+
 int migrate_vma_setup(struct migrate_vma *args);
 void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index b409c42bfe6f..f05f801b8404 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_device *mdevice)
 		HMM_DMIRROR_MEMORY_DEVICE_PRIVATE);
 }
 
-static enum migrate_vma_direction
+static enum migrate_vma_info
 dmirror_select_device(struct dmirror *dmirror)
 {
 	return (dmirror->mdevice->zone_device_type ==
diff --git a/mm/hmm.c b/mm/hmm.c
index 2f1e98c6b644..9220e6648636 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -21,6 +21,7 @@
 #include <linux/pagemap.h>
 #include <linux/leafops.h>
 #include <linux/hugetlb.h>
+#include <linux/migrate.h>
 #include <linux/memremap.h>
 #include <linux/sched/mm.h>
 #include <linux/jump_label.h>
@@ -28,15 +29,23 @@
 #include <linux/pci-p2pdma.h>
 #include <linux/mmu_notifier.h>
 #include <linux/memory_hotplug.h>
+#include <asm/tlbflush.h>
 
 #include "internal.h"
 
 struct hmm_vma_walk {
-	struct hmm_range	*range;
-	bool			*locked;
-	unsigned long		last;
-	unsigned long		end;
-	unsigned int		required_fault;
+	struct mmu_notifier_range	mmu_range;
+	/* Set once mmu_range is armed with mmu_notifier_invalidate_range_start() */
+	bool				mmu_armed;
+	struct vm_area_struct		*vma;
+	struct hmm_range		*range;
+	unsigned long			start;
+	unsigned long			end;
+	unsigned long			last;
+	/* fault and lock drop related fields */
+	bool				*mmlocked;
+	unsigned long			fault_end;
+	unsigned int			required_fault;
 };
 
 /*
@@ -95,7 +104,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
 
 	WARN_ON_ONCE(!required_fault);
 	hmm_vma_walk->last = addr;
-	hmm_vma_walk->end = end;
+	hmm_vma_walk->fault_end = end;
 	hmm_vma_walk->required_fault = required_fault;
 	return HMM_FAULT_PENDING;
 }
@@ -176,7 +185,7 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
 	required_fault =
 		hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0);
 	if (!walk->vma) {
-		if (required_fault)
+		if (required_fault || hmm_select_migrate(range))
 			return -EFAULT;
 		return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
 	}
@@ -400,6 +409,73 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 }
 #endif  /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
 
+static int hmm_vma_capture_migrate_range(unsigned long start,
+					 unsigned long end,
+					 struct mm_walk *walk)
+{
+	struct hmm_vma_walk *hmm_vma_walk = walk->private;
+	struct hmm_range *range = hmm_vma_walk->range;
+	struct vm_area_struct *vma = walk->vma;
+
+	if (!hmm_select_migrate(range))
+		return 0;
+
+	if (!hmm_vma_walk->vma) {
+		/* Same vma restrictions as migrate_vma_setup() */
+		if (!vma_is_mm_backed(vma) || vma_is_hugetlb(vma) ||
+		    vma_is_dax(vma))
+			return -EINVAL;
+		/*
+		 * migrate_vma covers the whole range and indexes src[] from
+		 * range->start, so a single vma must cover the whole range.
+		 */
+		if (vma->vm_start > range->start ||
+		    vma->vm_end < range->end)
+			return -ERANGE;
+		hmm_vma_walk->vma = vma;
+		hmm_vma_walk->start = start;
+		hmm_vma_walk->end = end;
+	}
+
+	/*
+	 * The invalidation is closed before faulting, see
+	 * hmm_range_fault_locked(), so (re)open it for each walk.
+	 */
+	if (!hmm_vma_walk->mmu_armed) {
+		mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRATE, 0,
+					      walk->vma->vm_mm, hmm_vma_walk->start,
+					      hmm_vma_walk->end, range->dev_private_owner);
+		mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range);
+		hmm_vma_walk->mmu_armed = true;
+	}
+
+	return 0;
+}
+
+static void hmm_vma_migrate_disarm(struct hmm_vma_walk *hmm_vma_walk)
+{
+	if (hmm_vma_walk->mmu_armed) {
+		mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range);
+		hmm_vma_walk->mmu_armed = false;
+	}
+}
+
+static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk)
+{
+	struct hmm_range *range = hmm_vma_walk->range;
+
+	/*
+	 * The migrate_vma path has the following initialized. For the fault
+	 * path fill them in, also when the walk failed before capturing the
+	 * vma, so migrate_hmm_range_setup() always sees the range.
+	 */
+	if (hmm_select_migrate(range) && range->notifier)
+		hmm_fill_migrate_vma(range,
+				     hmm_vma_walk->vma,
+				     range->start, range->end);
+	hmm_vma_migrate_disarm(hmm_vma_walk);
+}
+
 static int hmm_vma_walk_pmd(pmd_t *pmdp,
 			    unsigned long start,
 			    unsigned long end,
@@ -594,10 +670,13 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
 	struct hmm_vma_walk *hmm_vma_walk = walk->private;
 	struct hmm_range *range = hmm_vma_walk->range;
 	struct vm_area_struct *vma = walk->vma;
+	/* Migration without faulting moves pages regardless of VM_READ */
+	bool migrate_only = hmm_select_migrate(range) &&
+			    !(range->default_flags & HMM_PFN_REQ_FAULT);
 
 	if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) &&
-	    vma->vm_flags & VM_READ)
-		return 0;
+	    (vma->vm_flags & VM_READ || migrate_only))
+		return hmm_vma_capture_migrate_range(start, end, walk);
 
 	/*
 	 * vma ranges that don't have struct page backing them or map I/O
@@ -607,10 +686,11 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
 	 * allow write access either. HMM does not support architectures that
 	 * allow write without read.
 	 *
-	 * If a fault is requested for an unsupported range then it is a hard
-	 * failure.
+	 * If a fault or migration is requested for an unsupported range then
+	 * it is a hard failure.
 	 */
-	if (hmm_range_need_fault(hmm_vma_walk,
+	if (hmm_select_migrate(range) ||
+	    hmm_range_need_fault(hmm_vma_walk,
 				 range->hmm_pfns +
 					 ((start - range->start) >> PAGE_SHIFT),
 				 (end - start) >> PAGE_SHIFT, 0))
@@ -646,12 +726,12 @@ static int hmm_do_fault(struct mm_struct *mm,
 			struct hmm_vma_walk *hmm_vma_walk)
 {
 	unsigned long addr = hmm_vma_walk->last;
-	unsigned long end = hmm_vma_walk->end;
+	unsigned long end = hmm_vma_walk->fault_end;
 	unsigned int required_fault = hmm_vma_walk->required_fault;
 	unsigned int fault_flags = FAULT_FLAG_REMOTE;
 	struct vm_area_struct *vma;
 
-	if (hmm_vma_walk->locked)
+	if (hmm_vma_walk->mmlocked)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 
 	vma = vma_lookup(mm, addr);
@@ -670,8 +750,8 @@ static int hmm_do_fault(struct mm_struct *mm,
 		ret = handle_mm_fault(vma, addr, fault_flags, NULL);
 
 		if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
-			if (hmm_vma_walk->locked)    /* needed by sparse */
-				*hmm_vma_walk->locked = false;
+			if (hmm_vma_walk->mmlocked)    /* needed by sparse */
+				*hmm_vma_walk->mmlocked = false;
 			else
 				WARN_ON_ONCE(1);    /* broken fault handler */
 			return HMM_FAULT_UNLOCKED;
@@ -694,19 +774,45 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
 {
 	struct hmm_vma_walk hmm_vma_walk = {
 		.range = range,
-		.locked = locked,
+		.mmlocked = locked,
 		.last = range->start,
 	};
-	struct mm_struct *mm = range->notifier->mm;
+	/*
+	 *  Could be serving a device fault or come from migrate
+	 *  entry point. For the former we have not resolved the vma
+	 *  yet, and the latter we don't have a notifier (but have a vma).
+	 *
+	 */
+	struct mm_struct *mm = hmm_range_fault_mm(range);
 	int ret;
 
+	/* Migration is incompatible with mmap lock drop */
+	if (locked && hmm_select_migrate(range))
+		return -EINVAL;
+
+	/*
+	 * Migration takes its requests only from default_flags, so the input
+	 * pfns carry nothing. Clear them so that collected entries can be
+	 * told apart reliably during the walk and on rollback.
+	 */
+	if (hmm_select_migrate(range)) {
+		WARN_ON_ONCE(range->pfn_flags_mask);
+		WARN_ON_ONCE(range->dev_private_owner !=
+			     range->migrate->pgmap_owner);
+		memset(range->hmm_pfns, 0,
+		       ((range->end - range->start) >> PAGE_SHIFT) *
+		       sizeof(*range->hmm_pfns));
+	}
+
 	mmap_assert_locked(mm);
 
 	do {
 		/* If range is no longer valid force retry. */
-		if (mmu_interval_check_retry(range->notifier,
-					     range->notifier_seq))
-			return -EBUSY;
+		if (range->notifier && mmu_interval_check_retry(range->notifier,
+								range->notifier_seq)) {
+			ret =  -EBUSY;
+			break;
+		}
 		ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
 				      &hmm_walk_ops, &hmm_vma_walk);
 		/*
@@ -718,6 +824,13 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
 		 * now-present entries.
 		 */
 		if (ret == HMM_FAULT_PENDING) {
+			/*
+			 * Don't hold the migrate invalidation across a
+			 * possibly sleeping fault. Already collected entries
+			 * stay locked and referenced, and the next walk
+			 * reopens the invalidation.
+			 */
+			hmm_vma_migrate_disarm(&hmm_vma_walk);
 			ret = hmm_do_fault(mm, &hmm_vma_walk);
 			if (ret == HMM_FAULT_UNLOCKED) {
 				if (fatal_signal_pending(current))
@@ -732,6 +845,8 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
 		 * output, and all >= are still at their input values.
 		 */
 	} while (ret == -EBUSY);
+
+	hmm_vma_post_range_fault(&hmm_vma_walk);
 	return ret;
 }
 
-- 
2.55.0

[PATCH v17 04/12] mm/hmm: do the plumbing for HMM to participate in migration

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Do the preparations in hmm_range_fault() and pagewalk callbacks to
do the "collecting" part of migration, needed for migration.

These steps include locking for pmd/pte if migrating, and calling the
still dummy hmm_vma_handle_migrate_prepare_pmd() and
hmm_vma_handle_migrate_prepare()  functions in the pagewalk.

When doing migration, have to have pmd/pte table locked for the
duration of both hmm_vma_handle*() and hmm_vma_handle_migrate_prepare*()
walks, and unlock when handling the faults.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 mm/hmm.c | 342 +++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 295 insertions(+), 47 deletions(-)

diff --git a/mm/hmm.c b/mm/hmm.c
index 9220e6648636..854d2c090b20 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -42,6 +42,16 @@ struct hmm_vma_walk {
 	unsigned long			start;
 	unsigned long			end;
 	unsigned long			last;
+	/*
+	 * For migration we need pte/pmd locked for the handle_* and
+	 * prepare_* regions. While faulting we have to drop the locks
+	 * and start again. ptelocked and pmdlocked hold the state
+	 * and tell  if need to drop locks before faulting.
+	 * ptl is the lock held for pte or pmd.
+	 */
+	bool				ptelocked;
+	bool				pmdlocked;
+	spinlock_t			*ptl;
 	/* fault and lock drop related fields */
 	bool				*mmlocked;
 	unsigned long			fault_end;
@@ -63,6 +73,16 @@ struct hmm_vma_walk {
  */
 #define HMM_FAULT_UNLOCKED	-ENOLCK
 
+#define HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, locked)		\
+	WARN_ON_ONCE((hmm_vma_walk)->ptelocked != locked)
+
+#define HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, locked)		\
+	WARN_ON_ONCE((hmm_vma_walk)->pmdlocked != locked)
+
+#define HMM_ASSERT_UNLOCKED(hmm_vma_walk)			\
+	WARN_ON_ONCE((hmm_vma_walk)->ptelocked ||		\
+		     (hmm_vma_walk)->pmdlocked)
+
 enum {
 	HMM_NEED_FAULT = 1 << 0,
 	HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -76,14 +96,38 @@ enum {
 };
 
 static int hmm_pfns_fill(unsigned long addr, unsigned long end,
-			 struct hmm_range *range, unsigned long cpu_flags)
+			 struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags)
 {
+	struct hmm_range *range = hmm_vma_walk->range;
 	unsigned long i = (addr - range->start) >> PAGE_SHIFT;
+	enum migrate_vma_info minfo;
+	bool migrate = false;
+
+	minfo = hmm_select_migrate(range);
+	if (cpu_flags != HMM_PFN_ERROR) {
+		if (minfo && (vma_is_anonymous(hmm_vma_walk->vma))) {
+			cpu_flags |= HMM_PFN_MIGRATE;
+			migrate = true;
+		}
+	}
+
+	if (migrate && thp_migration_supported() &&
+	    (minfo & MIGRATE_VMA_SELECT_COMPOUND) &&
+	    IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
+	    IS_ALIGNED(end, HPAGE_PMD_SIZE) &&
+		end-addr == HPAGE_PMD_SIZE) {
+		range->hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS;
+		range->hmm_pfns[i] |= cpu_flags | HMM_PFN_COMPOUND;
+		addr += PAGE_SIZE;
+		i++;
+		cpu_flags = 0;
+	}
 
 	for (; addr < end; addr += PAGE_SIZE, i++) {
 		range->hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS;
 		range->hmm_pfns[i] |= cpu_flags;
 	}
+
 	return 0;
 }
 
@@ -103,6 +147,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
 	struct hmm_vma_walk *hmm_vma_walk = walk->private;
 
 	WARN_ON_ONCE(!required_fault);
+	HMM_ASSERT_UNLOCKED(hmm_vma_walk);
 	hmm_vma_walk->last = addr;
 	hmm_vma_walk->fault_end = end;
 	hmm_vma_walk->required_fault = required_fault;
@@ -187,11 +232,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
 	if (!walk->vma) {
 		if (required_fault || hmm_select_migrate(range))
 			return -EFAULT;
-		return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
+		return hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR);
 	}
-	if (required_fault)
+	if (required_fault) {
+		if (hmm_vma_walk->pmdlocked) {
+			spin_unlock(hmm_vma_walk->ptl);
+			hmm_vma_walk->pmdlocked = false;
+		}
 		return hmm_record_fault(addr, end, required_fault, walk);
-	return hmm_pfns_fill(addr, end, range, 0);
+	}
+	return hmm_pfns_fill(addr, end, hmm_vma_walk, 0);
 }
 
 static inline unsigned long hmm_pfn_flags_order(unsigned long order)
@@ -224,8 +274,13 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
 	cpu_flags = pmd_to_hmm_pfn_flags(range, pmd);
 	required_fault =
 		hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
-	if (required_fault)
+	if (required_fault) {
+		if (hmm_vma_walk->pmdlocked) {
+			spin_unlock(hmm_vma_walk->ptl);
+			hmm_vma_walk->pmdlocked = false;
+		}
 		return hmm_record_fault(addr, end, required_fault, walk);
+	}
 
 	pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
 	for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -305,6 +360,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 			goto fault;
 
 		if (softleaf_is_migration(entry)) {
+			/*
+			 * Someone else's migration: when migrating, wait in
+			 * handle_mm_fault() via hmm_do_fault(), outside the walk
+			 * and with the migrate invalidation closed.
+			 */
+			if (hmm_select_migrate(range))
+				goto fault;
+			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
 			pte_unmap(ptep);
 			hmm_vma_walk->last = addr;
 			migration_entry_wait(walk->mm, pmdp, addr);
@@ -312,7 +375,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 		}
 
 		/* Report error for everything else */
-		pte_unmap(ptep);
+
+		if (hmm_vma_walk->ptelocked) {
+			pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+			hmm_vma_walk->ptelocked = false;
+		} else {
+			pte_unmap(ptep);
+		}
+
 		return -EFAULT;
 	}
 
@@ -329,7 +399,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 	if (!vm_normal_page(walk->vma, addr, pte) &&
 	    !is_zero_pfn(pte_pfn(pte))) {
 		if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) {
-			pte_unmap(ptep);
+			if (hmm_vma_walk->ptelocked) {
+				pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+				hmm_vma_walk->ptelocked = false;
+			} else {
+				pte_unmap(ptep);
+			}
+
 			return -EFAULT;
 		}
 		new_pfn_flags = HMM_PFN_ERROR;
@@ -342,7 +418,12 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 	return 0;
 
 fault:
-	pte_unmap(ptep);
+	if (hmm_vma_walk->ptelocked) {
+		pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+		hmm_vma_walk->ptelocked = false;
+	} else {
+		pte_unmap(ptep);
+	}
 	/* Fault any virtual address we were asked to fault */
 	return hmm_record_fault(addr, end, required_fault, walk);
 }
@@ -386,13 +467,18 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 	required_fault = hmm_range_need_fault(hmm_vma_walk, hmm_pfns,
 					      npages, 0);
 	if (required_fault) {
-		if (softleaf_is_device_private(entry))
+		if (softleaf_is_device_private(entry)) {
+			if (hmm_vma_walk->pmdlocked) {
+				spin_unlock(hmm_vma_walk->ptl);
+				hmm_vma_walk->pmdlocked = false;
+			}
 			return hmm_record_fault(addr, end, required_fault, walk);
+		}
 		else
 			return -EFAULT;
 	}
 
-	return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+	return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
 }
 #else
 static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
@@ -400,15 +486,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 				     pmd_t pmd)
 {
 	struct hmm_vma_walk *hmm_vma_walk = walk->private;
-	struct hmm_range *range = hmm_vma_walk->range;
 	unsigned long npages = (end - start) >> PAGE_SHIFT;
 
 	if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0))
 		return -EFAULT;
-	return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+	return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
 }
 #endif  /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
 
+#ifdef CONFIG_DEVICE_MIGRATION
+static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
+					      pmd_t *pmdp,
+					      unsigned long start,
+					      unsigned long end,
+					      unsigned long *hmm_pfn)
+{
+	// TODO: implement migration entry insertion
+	return 0;
+}
+
+static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
+					  pmd_t *pmdp,
+					  pte_t *ptep,
+					  unsigned long addr,
+					  unsigned long *hmm_pfn,
+					  bool *unmapped)
+{
+	// TODO: implement migration entry insertion
+	return 0;
+}
+
+#else
+static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
+					      pmd_t *pmdp,
+					      unsigned long start,
+					      unsigned long end,
+					      unsigned long *hmm_pfn)
+{
+	return 0;
+}
+
+static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
+					  pmd_t *pmdp,
+					  pte_t *ptep,
+					  unsigned long addr,
+					  unsigned long *hmm_pfn,
+					  bool *unmapped)
+{
+	return 0;
+}
+
+#endif
+
 static int hmm_vma_capture_migrate_range(unsigned long start,
 					 unsigned long end,
 					 struct mm_walk *walk)
@@ -483,46 +612,124 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 {
 	struct hmm_vma_walk *hmm_vma_walk = walk->private;
 	struct hmm_range *range = hmm_vma_walk->range;
-	unsigned long *hmm_pfns =
-		&range->hmm_pfns[(start - range->start) >> PAGE_SHIFT];
 	unsigned long npages = (end - start) >> PAGE_SHIFT;
+	struct mm_struct *mm = walk->vma->vm_mm;
+	unsigned long *hmm_pfns, *hmm_pfns_start;
+	enum migrate_vma_info minfo;
 	unsigned long addr = start;
+	bool unmapped = false;
+	unsigned long i;
 	pte_t *ptep;
 	pmd_t pmd;
+	int r = 0;
 
+	minfo = hmm_select_migrate(range);
+	hmm_pfns_start = &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT];
 again:
-	pmd = pmdp_get_lockless(pmdp);
-	if (pmd_none(pmd))
-		return hmm_vma_walk_hole(start, end, -1, walk);
+	hmm_pfns = &range->hmm_pfns[(addr - range->start) >> PAGE_SHIFT];
+	hmm_vma_walk->ptelocked = false;
+	hmm_vma_walk->pmdlocked = false;
+
+	if (minfo) {
+		hmm_vma_walk->ptl = pmd_lock(mm, pmdp);
+		hmm_vma_walk->pmdlocked = true;
+		pmd = pmdp_get(pmdp);
+	} else
+		pmd = pmdp_get_lockless(pmdp);
+
+	if (pmd_none(pmd)) {
+		r = hmm_vma_walk_hole(start, end, -1, walk);
+
+		if (hmm_vma_walk->pmdlocked) {
+			spin_unlock(hmm_vma_walk->ptl);
+			hmm_vma_walk->pmdlocked = false;
+		}
+		return r;
+	}
 
 	if (thp_migration_supported() && pmd_is_migration_entry(pmd)) {
-		if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) {
+		unsigned int required_fault =
+			hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0);
+
+		if (hmm_vma_walk->pmdlocked) {
+			spin_unlock(hmm_vma_walk->ptl);
+			hmm_vma_walk->pmdlocked = false;
+		}
+
+		if (required_fault) {
+			/*
+			 * Someone else's migration: when migrating, wait in
+			 * handle_mm_fault() via hmm_do_fault(), outside the walk
+			 * and with the migrate invalidation closed.
+			 */
+			if (minfo)
+				return hmm_record_fault(addr, end, required_fault, walk);
 			hmm_vma_walk->last = addr;
 			pmd_migration_entry_wait(walk->mm, pmdp);
 			return -EBUSY;
 		}
-		return hmm_pfns_fill(start, end, range, 0);
+		for (i = 0; start < end; start += PAGE_SIZE, i++)
+			hmm_pfns_start[i] &= HMM_PFN_INOUT_FLAGS;
+
+		return 0;
 	}
 
-	if (!pmd_present(pmd))
-		return hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns,
-						 pmd);
+	if (pmd_trans_huge(pmd) || !pmd_present(pmd)) {
+		if (!pmd_present(pmd)) {
+			r = hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns_start,
+						      pmd);
+			// If not migrating we are done
+			if (r || !minfo) {
+				if (hmm_vma_walk->pmdlocked) {
+					spin_unlock(hmm_vma_walk->ptl);
+					hmm_vma_walk->pmdlocked = false;
+				}
+				return r;
+			}
+		}
 
-	if (pmd_trans_huge(pmd)) {
-		/*
-		 * No need to take pmd_lock here, even if some other thread
-		 * is splitting the huge pmd we will get that event through
-		 * mmu_notifier callback.
-		 *
-		 * So just read pmd value and check again it's a transparent
-		 * huge or device mapping one and compute corresponding pfn
-		 * values.
-		 */
-		pmd = pmdp_get_lockless(pmdp);
-		if (!pmd_trans_huge(pmd))
-			goto again;
+		if (pmd_trans_huge(pmd)) {
+			/*
+			 * No need to take pmd_lock here if not migrating,
+			 * even if some other thread is splitting the huge
+			 * pmd we will get that event through mmu_notifier callback.
+			 *
+			 * So just read pmd value and check again it's a transparent
+			 * huge or device mapping one and compute corresponding pfn
+			 * values.
+			 */
+
+			if (!minfo) {
+				pmd = pmdp_get_lockless(pmdp);
+				if (!pmd_trans_huge(pmd))
+					goto again;
+			}
+
+			r = hmm_vma_handle_pmd(walk, start, end, hmm_pfns_start, pmd);
+
+			// If not migrating we are done
+			if (r || !minfo) {
+				if (hmm_vma_walk->pmdlocked) {
+					spin_unlock(hmm_vma_walk->ptl);
+					hmm_vma_walk->pmdlocked = false;
+				}
+				return r;
+			}
+		}
+
+		r = hmm_vma_handle_migrate_prepare_pmd(walk, pmdp, start, end, hmm_pfns_start);
+
+		if (hmm_vma_walk->pmdlocked) {
+			spin_unlock(hmm_vma_walk->ptl);
+			hmm_vma_walk->pmdlocked = false;
+		}
+
+		return r;
+	}
 
-		return hmm_vma_handle_pmd(walk, addr, end, hmm_pfns, pmd);
+	if (hmm_vma_walk->pmdlocked) {
+		spin_unlock(hmm_vma_walk->ptl);
+		hmm_vma_walk->pmdlocked = false;
 	}
 
 	/*
@@ -532,24 +739,65 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 	 * recover.
 	 */
 	if (pmd_bad(pmd)) {
-		if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0))
+		if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0))
 			return -EFAULT;
-		return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+		return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
 	}
 
-	ptep = pte_offset_map(pmdp, addr);
-	if (!ptep)
+	if (minfo) {
+		ptep = pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl);
+		if (ptep)
+			hmm_vma_walk->ptelocked = true;
+	} else {
+		ptep = pte_offset_map(pmdp, addr);
+	}
+	if (!ptep) {
+		addr = start;
 		goto again;
+	}
+
 	for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) {
-		int r;
 
 		r = hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns);
 		if (r) {
-			/* hmm_vma_handle_pte() did pte_unmap() */
+			/* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */
+			if (unmapped)
+				flush_tlb_range(walk->vma, start, addr);
 			return r;
 		}
+
+		r = hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &unmapped);
+		if (r == -EAGAIN) {
+			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
+			if (unmapped) {
+				flush_tlb_range(walk->vma, start, addr);
+				unmapped = false;
+			}
+			goto again;
+		}
+		if (r) {
+			/*
+			 * Splitting the folio failed; migrate_vma_split_folio()
+			 * already dropped the PTE lock. Don't migrate the rest
+			 * of the pmd range; addr keeps its valid pfn.
+			 */
+			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
+			if (unmapped)
+				flush_tlb_range(walk->vma, start, addr);
+			return 0;
+		}
+	}
+
+	if (unmapped)
+		flush_tlb_range(walk->vma, start, addr);
+
+	if (hmm_vma_walk->ptelocked) {
+		pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl);
+		hmm_vma_walk->ptelocked = false;
+	} else {
+		pte_unmap(ptep - 1);
 	}
-	pte_unmap(ptep - 1);
+
 	return 0;
 }
 
@@ -696,7 +944,7 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
 				 (end - start) >> PAGE_SHIFT, 0))
 		return -EFAULT;
 
-	hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+	hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
 
 	/* Skip this vma and continue processing the next vma. */
 	return 1;
@@ -833,9 +1081,9 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
 			hmm_vma_migrate_disarm(&hmm_vma_walk);
 			ret = hmm_do_fault(mm, &hmm_vma_walk);
 			if (ret == HMM_FAULT_UNLOCKED) {
-				if (fatal_signal_pending(current))
-					return -EINTR;
-				return -EBUSY;
+				ret = fatal_signal_pending(current) ? -EINTR :
+					-EBUSY;
+				break;
 			}
 		}
 		/*
-- 
2.55.0

[PATCH v17 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Implement the needed hmm_vma_handle_migrate_prepare() function
which is mostly carried over from migrate_device.c's
migrate_vma_collect_pmd() function.

Also implement the migrate_vma_split_folio(), for splitting
pte mapped large folios. It is also mostly from migrate_device.c,
with care taken to reference folio before releasing page table
lock.

With HMM pagewalk based migration, the idea is that
hmm_vma_handle_*() are responsible for faulting,
and the pfn collecting part. hmm_vma_handle_migrate_prepare*()
do the migration decisions (with HMM_PFN_MIGRATE), possibly split
folios, and insert migration ptes/pmds.

Collected folios stay locked while hmm_range_fault() may still fault
other pages of the range. This cannot self-deadlock: only anonymous
folios are collected, so no page cache or invalidate_lock ordering is
involved; faults are confined to the single vma, where a non-KSM
anonymous page is mapped only once (and that mapping is now our
migration entry); and when migrating only the faulting pte is faulted,
never our own entries. A KSM page's second mapping just fails the
collection trylock, and a swapcache folio still referenced by other
swap entries is not collected. Reclaim and direct compaction under the
fault only trylock folio locks.

HMM pagewalk based migration is enabled in later commit, for
now hmm_select_migrate() returns 0.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 mm/hmm.c | 294 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 293 insertions(+), 1 deletion(-)

diff --git a/mm/hmm.c b/mm/hmm.c
index 854d2c090b20..ec9280ef2682 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -495,6 +495,71 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 #endif  /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
 
 #ifdef CONFIG_DEVICE_MIGRATION
+/**
+ * migrate_vma_split_folio() - Helper function to split a THP folio
+ * @folio: the folio to split
+ * @fault_page: struct page associated with the fault if any
+ * @hmm_vma_walk: walk in progress
+ * @ptep: pte_t * for unmap and unlock ptl
+ *
+ * Always drops the pte lock associated with @ptep.
+ *
+ * Returns 0 on success
+ */
+static int migrate_vma_split_folio(struct folio *folio,
+				   struct page *fault_page,
+				   struct hmm_vma_walk *hmm_vma_walk,
+				   pte_t *ptep)
+{
+	int ret;
+	struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
+	struct folio *new_fault_folio = NULL;
+
+	/*
+	 * Take the reference while still holding the PTE lock, so the folio
+	 * cannot be freed once the lock is dropped. folio_lock() may sleep,
+	 * so it can only be taken after dropping the PTE spinlock. The fault
+	 * folio is already referenced and locked by do_swap_page().
+	 */
+	if (folio != fault_folio)
+		folio_get(folio);
+
+	pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+	hmm_vma_walk->ptelocked = false;
+
+	if (folio != fault_folio)
+		folio_lock(folio);
+
+	ret = split_folio(folio);
+	if (ret) {
+		if (folio != fault_folio) {
+			folio_unlock(folio);
+			folio_put(folio);
+		}
+		return ret;
+	}
+
+	new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
+
+	/*
+	 * Ensure the lock is held on the correct
+	 * folio after the split
+	 */
+	if (!new_fault_folio) {
+		folio_unlock(folio);
+		folio_put(folio);
+	} else if (folio != new_fault_folio) {
+		if (new_fault_folio != fault_folio) {
+			folio_get(new_fault_folio);
+			folio_lock(new_fault_folio);
+		}
+		folio_unlock(folio);
+		folio_put(folio);
+	}
+
+	return 0;
+}
+
 static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
 					      pmd_t *pmdp,
 					      unsigned long start,
@@ -505,6 +570,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
 	return 0;
 }
 
+/*
+ * Install migration entries if migration requested, either from fault
+ * or migrate paths.
+ *
+ */
 static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
 					  pmd_t *pmdp,
 					  pte_t *ptep,
@@ -512,8 +582,230 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
 					  unsigned long *hmm_pfn,
 					  bool *unmapped)
 {
-	// TODO: implement migration entry insertion
+	struct hmm_vma_walk *hmm_vma_walk = walk->private;
+	struct hmm_range *range = hmm_vma_walk->range;
+	struct migrate_vma *migrate = range->migrate;
+	struct mm_struct *mm = walk->vma->vm_mm;
+	struct folio *fault_folio = NULL;
+	enum migrate_vma_info minfo;
+	struct dev_pagemap *pgmap;
+	bool anon_exclusive;
+	struct folio *folio;
+	unsigned long pfn;
+	struct page *page;
+	softleaf_t entry;
+	pte_t pte, swp_pte;
+	bool writable = false;
+
+	// Do we want to migrate at all?
+	minfo = hmm_select_migrate(range);
+	if (!minfo)
+		return 0;
+
+	WARN_ON_ONCE(!migrate);
+	HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true);
+
+	fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
+
+	pte = ptep_get(ptep);
+
+	if (pte_none(pte)) {
+		if (vma_is_anonymous(walk->vma)) {
+			*hmm_pfn &= HMM_PFN_INOUT_FLAGS;
+			*hmm_pfn |= HMM_PFN_MIGRATE;
+			goto out;
+		}
+	}
+
+	/*
+	 * NUMA hinting (protnone) entries are not reported valid, but the
+	 * page can still be migrated as is.
+	 */
+	if (!(hmm_pfn[0] & HMM_PFN_VALID) &&
+	    !(pte_present(pte) && pte_protnone(pte)))
+		goto out;
+
+	if (!pte_present(pte)) {
+		/*
+		 * Only care about unaddressable device page special
+		 * page table entry. Other special swap entries are not
+		 * migratable, and we ignore regular swapped page.
+		 */
+		entry = softleaf_from_pte(pte);
+		if (!softleaf_is_device_private(entry))
+			goto out;
+
+		if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
+			goto out;
+
+		page = softleaf_to_page(entry);
+		folio = page_folio(page);
+		if (folio->pgmap->owner != migrate->pgmap_owner)
+			goto out;
+
+		if (folio_test_large(folio)) {
+			int ret;
+
+			ret = migrate_vma_split_folio(folio,
+						      migrate->fault_page,
+						      hmm_vma_walk,
+						      ptep);
+			if (ret)
+				goto out_error;
+			return -EAGAIN;
+		}
+
+		pfn = page_to_pfn(page);
+		if (softleaf_is_device_private_write(entry))
+			writable = true;
+	} else {
+		pfn = pte_pfn(pte);
+		if (is_zero_pfn(pfn) &&
+		    (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+			/* Like pte_none(): only anonymous memory can be populated */
+			if (!vma_is_anonymous(walk->vma))
+				goto out;
+			*hmm_pfn &= HMM_PFN_INOUT_FLAGS;
+			*hmm_pfn |= HMM_PFN_MIGRATE;
+			goto out;
+		}
+		page = vm_normal_page(walk->vma, addr, pte);
+		if (page && !is_zone_device_page(page) &&
+		    !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+			goto out;
+		} else if (page && is_device_coherent_page(page)) {
+			pgmap = page_pgmap(page);
+
+			if (!(minfo &
+			      MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
+			    pgmap->owner != migrate->pgmap_owner)
+				goto out;
+		}
+
+		folio = page ? page_folio(page) : NULL;
+		if (folio && folio_test_large(folio)) {
+			int ret;
+
+			ret = migrate_vma_split_folio(folio,
+						      migrate->fault_page,
+						      hmm_vma_walk,
+						      ptep);
+			if (ret)
+				goto out_error;
+			return -EAGAIN;
+		}
+
+		writable = pte_write(pte);
+	}
+
+	/* Only anonymous memory can be migrated to device memory */
+	if (!page || !folio_test_anon(page_folio(page)))
+		goto out;
+
+	/*
+	 * By getting a reference on the folio we pin it and that blocks
+	 * any kind of migration. Side effect is that it "freezes" the
+	 * pte.
+	 *
+	 * We drop this reference after isolating the folio from the lru
+	 * for non device folio (device folio are not on the lru and thus
+	 * can't be dropped from it).
+	 */
+	folio = page_folio(page);
+	folio_get(folio);
+
+	/*
+	 * We rely on folio_trylock() to avoid deadlock between
+	 * concurrent migrations where each is waiting on the others
+	 * folio lock. If we can't immediately lock the folio we fail this
+	 * migration as it is only best effort anyway.
+	 *
+	 * If we can lock the folio it's safe to set up a migration entry
+	 * now. In the common case where the folio is mapped once in a
+	 * single process setting up the migration entry now is an
+	 * optimisation to avoid walking the rmap later with
+	 * try_to_migrate().
+	 */
+
+	if (fault_folio == folio || folio_trylock(folio)) {
+		/*
+		 * A swapcache folio still referenced by other swap entries
+		 * can't be migrated (see migrate_device_pages()), and faulting
+		 * one of those entries later in the walk, with no retry
+		 * allowed, would block on the lock we hold.
+		 */
+		if (folio_test_swapcache(folio) && !folio_free_swap(folio)) {
+			if (fault_folio != folio)
+				folio_unlock(folio);
+			folio_put(folio);
+			goto out;
+		}
+
+		anon_exclusive = folio_test_anon(folio) &&
+			PageAnonExclusive(page);
+
+		if (pte_present(pte))
+			flush_cache_page(walk->vma, addr, pfn);
+
+		if (anon_exclusive) {
+			pte = ptep_clear_flush(walk->vma, addr, ptep);
+
+			if (folio_try_share_anon_rmap_pte(folio, page)) {
+				set_pte_at(mm, addr, ptep, pte);
+				if (fault_folio != folio)
+					folio_unlock(folio);
+				folio_put(folio);
+				goto out;
+			}
+		} else {
+			pte = ptep_get_and_clear(mm, addr, ptep);
+		}
+
+		if (pte_present(pte) && pte_dirty(pte))
+			folio_mark_dirty(folio);
+
+		/* Setup special migration page table entry */
+		if (writable)
+			entry = make_writable_migration_entry(pfn);
+		else if (anon_exclusive)
+			entry = make_readable_exclusive_migration_entry(pfn);
+		else
+			entry = make_readable_migration_entry(pfn);
+
+		if (pte_present(pte)) {
+			if (pte_young(pte))
+				entry = make_migration_entry_young(entry);
+			if (pte_dirty(pte))
+				entry = make_migration_entry_dirty(entry);
+		}
+
+		swp_pte = swp_entry_to_pte(entry);
+		if (pte_present(pte)) {
+			if (pte_soft_dirty(pte))
+				swp_pte = pte_swp_mksoft_dirty(swp_pte);
+			if (pte_uffd(pte))
+				swp_pte = pte_swp_mkuffd(swp_pte);
+		} else {
+			if (pte_swp_soft_dirty(pte))
+				swp_pte = pte_swp_mksoft_dirty(swp_pte);
+			if (pte_swp_uffd(pte))
+				swp_pte = pte_swp_mkuffd(swp_pte);
+		}
+
+		set_pte_at(mm, addr, ptep, swp_pte);
+		folio_remove_rmap_pte(folio, page, walk->vma);
+		folio_put(folio);
+		*hmm_pfn |= HMM_PFN_VALID | HMM_PFN_MIGRATE;
+		if (pte_present(pte))
+			*unmapped = true;
+	} else {
+		folio_put(folio);
+	}
+out:
 	return 0;
+out_error:
+	return -EFAULT;
 }
 
 #else
-- 
2.55.0

[PATCH v17 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Implement the needed hmm_vma_handle_migrate_prepare_pmd() function
which is mostly carried over from migrate_device.c's
migrate_vma_collect_huge_pmd() function.

With HMM pagewalk based migration, the idea is that
hmm_vma_handle_*() are responsible for faulting, and the pfn
collecting part. hmm_vma_handle_migrate_prepare*() do the
migration decisions (with HMM_PFN_MIGRATE), possibly split
folios, and insert migration ptes/pmds.

Locking follows the same rules as at the pte level (see the previous
commit): the folio is only trylocked, a PMD-mapped THP is collected
whole behind its pmd migration entry (its only mapping in the vma), and
a swapcache folio still referenced by other swap entries is not
collected. On a trylock or split failure the valid pfns are kept and
nothing is migrated.

HMM pagewalk based migration is enabled in later commit, for now
hmm_select_migrate() returns 0.

Also avoid the problems present in the migrate_vma_collect_huge_pmd()
split fallback path this HMM path supersedes:

      - extra refcount for fault_folio makes split_folio() fail
      - wrong folio unlocked after split_folio()

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 mm/hmm.c | 183 +++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 152 insertions(+), 31 deletions(-)

diff --git a/mm/hmm.c b/mm/hmm.c
index ec9280ef2682..cbde004fd778 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -496,7 +496,33 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 
 #ifdef CONFIG_DEVICE_MIGRATION
 /**
- * migrate_vma_split_folio() - Helper function to split a THP folio
+ * __migrate_vma_split_folio() - split a locked folio for migration
+ * @folio: the folio to split, locked and referenced by the caller
+ * @fault_page: fault page if any
+ *
+ * The fault folio is split at @fault_page, so the caller's lock and reference
+ * stay on the folio containing @fault_page. For any other folio our lock and
+ * reference are dropped, whether or not the split succeeds.
+ *
+ * Returns 0 on success.
+ */
+static int __migrate_vma_split_folio(struct folio *folio,
+				     struct page *fault_page)
+{
+	int ret;
+
+	if (fault_page && folio == page_folio(fault_page))
+		return split_huge_page(fault_page);
+
+	/* Someone else may have split it while we waited for the lock */
+	ret = folio_test_large(folio) ? split_folio(folio) : 0;
+	folio_unlock(folio);
+	folio_put(folio);
+	return ret;
+}
+
+/**
+ * migrate_vma_split_folio() - drop the pte lock and split a THP folio
  * @folio: the folio to split
  * @fault_page: struct page associated with the fault if any
  * @hmm_vma_walk: walk in progress
@@ -511,9 +537,7 @@ static int migrate_vma_split_folio(struct folio *folio,
 				   struct hmm_vma_walk *hmm_vma_walk,
 				   pte_t *ptep)
 {
-	int ret;
 	struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
-	struct folio *new_fault_folio = NULL;
 
 	/*
 	 * Take the reference while still holding the PTE lock, so the folio
@@ -530,44 +554,136 @@ static int migrate_vma_split_folio(struct folio *folio,
 	if (folio != fault_folio)
 		folio_lock(folio);
 
-	ret = split_folio(folio);
-	if (ret) {
-		if (folio != fault_folio) {
+	return __migrate_vma_split_folio(folio, fault_page);
+}
+
+static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
+					      pmd_t *pmdp,
+					      unsigned long start,
+					      unsigned long end,
+					      unsigned long *hmm_pfn)
+{
+	struct hmm_vma_walk *hmm_vma_walk = walk->private;
+	struct hmm_range *range = hmm_vma_walk->range;
+	struct migrate_vma *migrate = range->migrate;
+	struct folio *fault_folio = NULL;
+	enum migrate_vma_info minfo;
+	struct folio *folio;
+	unsigned long i;
+	int r = 0;
+
+	// Do we want to migrate at all?
+	minfo = hmm_select_migrate(range);
+	if (!minfo)
+		return r;
+
+	WARN_ON_ONCE(!migrate);
+	HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true);
+
+	fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
+
+	if (pmd_none(*pmdp))
+		return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
+
+	/* Like at the pte level, NUMA hinting (protnone) pmds are migrated */
+	if (!(hmm_pfn[0] & HMM_PFN_VALID) &&
+	    !(pmd_trans_huge(*pmdp) && pmd_protnone(*pmdp)))
+		goto out;
+
+	if (pmd_trans_huge(*pmdp)) {
+		if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM))
+			goto out;
+
+		folio = pmd_folio(*pmdp);
+		if (is_huge_zero_folio(folio))
+			return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
+
+	} else if (!pmd_present(*pmdp)) {
+		const softleaf_t entry = softleaf_from_pmd(*pmdp);
+
+		if (!softleaf_is_device_private(entry))
+			goto out;
+
+		if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
+			goto out;
+
+		folio = softleaf_to_folio(entry);
+		if (folio->pgmap->owner != migrate->pgmap_owner)
+			goto out;
+	} else {
+		hmm_vma_walk->last = start;
+		return -EBUSY;
+	}
+
+	folio_get(folio);
+
+	/* Can't lock it: not migrated, the valid pfns stay as reported */
+	if (folio != fault_folio && unlikely(!folio_trylock(folio))) {
+		folio_put(folio);
+		return 0;
+	}
+
+	/* Swapcache folio still referenced elsewhere: see the pte level */
+	if (folio_test_swapcache(folio) && !folio_free_swap(folio)) {
+		if (folio != fault_folio)
 			folio_unlock(folio);
-			folio_put(folio);
+		folio_put(folio);
+		return 0;
+	}
+
+	if (thp_migration_supported() &&
+	    (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
+	    (IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
+	     IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
+		struct page_vma_mapped_walk pvmw = {
+			.ptl = hmm_vma_walk->ptl,
+			.address = start,
+			.pmd = pmdp,
+			.vma = walk->vma,
+		};
+
+		hmm_pfn[0] |= HMM_PFN_MIGRATE | HMM_PFN_COMPOUND;
+
+		r = set_pmd_migration_entry(&pvmw, folio_page(folio, 0));
+		if (r) {
+			hmm_pfn[0] &= ~(HMM_PFN_MIGRATE | HMM_PFN_COMPOUND);
+			goto split;	/* fall back to splitting the pmd */
 		}
-		return ret;
+		/* Collected; a protnone pmd was not reported valid before */
+		hmm_pfn[0] |= HMM_PFN_VALID;
+		for (i = 1, start += PAGE_SIZE; start < end; start += PAGE_SIZE, i++)
+			hmm_pfn[i] &= HMM_PFN_INOUT_FLAGS;
+
+	} else {
+		goto split;		/* fall back to splitting the pmd */
 	}
 
-	new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
+out:
+	return r;
+
+split:
+	spin_unlock(hmm_vma_walk->ptl);
+	hmm_vma_walk->pmdlocked = false;
 
 	/*
-	 * Ensure the lock is held on the correct
-	 * folio after the split
+	 * folio_get() above took an extra reference. For the fault folio the
+	 * caller still holds a reference and the lock, which is the
+	 * precondition of __migrate_vma_split_folio(), so drop the extra one.
 	 */
-	if (!new_fault_folio) {
-		folio_unlock(folio);
+	if (folio == fault_folio)
 		folio_put(folio);
-	} else if (folio != new_fault_folio) {
-		if (new_fault_folio != fault_folio) {
-			folio_get(new_fault_folio);
-			folio_lock(new_fault_folio);
-		}
-		folio_unlock(folio);
-		folio_put(folio);
-	}
 
-	return 0;
-}
+	/* Split failed: not migrated, the valid pfns stay as reported */
+	if (__migrate_vma_split_folio(folio, migrate->fault_page))
+		return 0;
 
-static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
-					      pmd_t *pmdp,
-					      unsigned long start,
-					      unsigned long end,
-					      unsigned long *hmm_pfn)
-{
-	// TODO: implement migration entry insertion
-	return 0;
+	/* Re-walked as PTEs from start: drop the pmd-level output */
+	for (i = 0; i < (end - start) >> PAGE_SHIFT; i++)
+		hmm_pfn[i] &= HMM_PFN_INOUT_FLAGS;
+
+	hmm_vma_walk->last = start;
+	return -EBUSY;
 }
 
 /*
@@ -1016,6 +1132,11 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 			hmm_vma_walk->pmdlocked = false;
 		}
 
+		/*
+		 * hmm_vma_handle_migrate_prepare_pmd() splits the huge pmd in
+		 * place when needed and returns -EBUSY to re-walk the range as
+		 * PTEs.
+		 */
 		return r;
 	}
 
-- 
2.55.0

[PATCH v17 07/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Add calls to lazy_mmu_mode_enable() and lazy_mmu_mode_disable()
while doing migration.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 mm/hmm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/mm/hmm.c b/mm/hmm.c
index cbde004fd778..690b73908f34 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -377,6 +377,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 		/* Report error for everything else */
 
 		if (hmm_vma_walk->ptelocked) {
+			lazy_mmu_mode_disable();
 			pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
 			hmm_vma_walk->ptelocked = false;
 		} else {
@@ -400,6 +401,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 	    !is_zero_pfn(pte_pfn(pte))) {
 		if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) {
 			if (hmm_vma_walk->ptelocked) {
+				lazy_mmu_mode_disable();
 				pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
 				hmm_vma_walk->ptelocked = false;
 			} else {
@@ -419,6 +421,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 
 fault:
 	if (hmm_vma_walk->ptelocked) {
+		lazy_mmu_mode_disable();
 		pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
 		hmm_vma_walk->ptelocked = false;
 	} else {
@@ -548,6 +551,7 @@ static int migrate_vma_split_folio(struct folio *folio,
 	if (folio != fault_folio)
 		folio_get(folio);
 
+	lazy_mmu_mode_disable();
 	pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
 	hmm_vma_walk->ptelocked = false;
 
@@ -1159,8 +1163,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 
 	if (minfo) {
 		ptep = pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl);
-		if (ptep)
+		if (ptep) {
+			lazy_mmu_mode_enable();
 			hmm_vma_walk->ptelocked = true;
+		}
 	} else {
 		ptep = pte_offset_map(pmdp, addr);
 	}
@@ -1205,6 +1211,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 		flush_tlb_range(walk->vma, start, addr);
 
 	if (hmm_vma_walk->ptelocked) {
+		lazy_mmu_mode_disable();
 		pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl);
 		hmm_vma_walk->ptelocked = false;
 	} else {
-- 
2.55.0

[PATCH v17 08/12] mm/hmm: implement rollback for device page migration in HMM pagewalk

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

During the migration pagewalk, the PTE table could be cleared and/or
changed into PMD leaf or even another PTE table while dropped locks.

In these cases the possibly inserted migration ptes are gone. We have
to however undo the collecting done so far, so unlock the folios and
drop reference taken.

During the pagewalk we notice such scenarios if going to recollect a pfn
but have already committed to migrate the entry with HMM_PFN_MIGRATE, in
which case do rollback. Rollback clears (HMM_PFN_VALID | HMM_PFN_MIGRATE)
which is safe, as migration is entirely driven by default_flags.

If we encounter migration ptes they are just skipped to allow for restart
own walks.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 include/linux/hmm.h |  22 ++++++++
 mm/hmm.c            | 127 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 148 insertions(+), 1 deletion(-)

diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 4f56f3419cb4..b08ebc1343dd 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
 	return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F;
 }
 
+/*
+ * hmm_pfn_collected() - is this pfn entry prepared for migration ?
+ * If collected the folio's refcount is increased and the folio
+ * is locked.
+ */
+static inline bool hmm_pfn_collected(unsigned long hmm_pfn)
+{
+	return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) ==
+		(HMM_PFN_VALID | HMM_PFN_MIGRATE);
+}
+
+/*
+ * hmm_pfn_rollback_collected() - undoes the collection of hmm_pfn
+ *
+ * Note for total rollback the folio's refcount has to be put
+ * and folio has to be unlocked.
+ */
+static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_pfn)
+{
+	return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND);
+}
+
 /*
  * struct hmm_range - track invalidation lock on virtual address range
  *
diff --git a/mm/hmm.c b/mm/hmm.c
index 690b73908f34..1c8a3fa29c4e 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -95,6 +95,11 @@ enum {
 			      HMM_PFN_P2PDMA_BUS,
 };
 
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+						    unsigned long start,
+						    unsigned long end,
+						    unsigned long *hmm_pfn);
+
 static int hmm_pfns_fill(unsigned long addr, unsigned long end,
 			 struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags)
 {
@@ -111,6 +116,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
 		}
 	}
 
+	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->hmm_pfns[i]);
+
 	if (migrate && thp_migration_supported() &&
 	    (minfo & MIGRATE_VMA_SELECT_COMPOUND) &&
 	    IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
@@ -239,6 +246,9 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
 			spin_unlock(hmm_vma_walk->ptl);
 			hmm_vma_walk->pmdlocked = false;
 		}
+		/* Entries collected from a vanished page table are gone */
+		hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end,
+							hmm_pfns);
 		return hmm_record_fault(addr, end, required_fault, walk);
 	}
 	return hmm_pfns_fill(addr, end, hmm_vma_walk, 0);
@@ -274,6 +284,9 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
 	cpu_flags = pmd_to_hmm_pfn_flags(range, pmd);
 	required_fault =
 		hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
+	/* Entries collected from a vanished page table are gone */
+	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
+						end, hmm_pfns);
 	if (required_fault) {
 		if (hmm_vma_walk->pmdlocked) {
 			spin_unlock(hmm_vma_walk->ptl);
@@ -331,6 +344,15 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 	if (!pte_present(pte)) {
 		const softleaf_t entry = softleaf_from_pte(pte);
 
+		/*
+		 * Our own migration entry, collected earlier in this walk:
+		 * keep the collected pfn, don't fault or roll it back.
+		 */
+		if (softleaf_is_migration(entry) && hmm_select_migrate(range) &&
+		    hmm_pfn_collected(*hmm_pfn) &&
+		    hmm_pfn_to_page(*hmm_pfn) == softleaf_to_page(entry))
+			return 0;
+
 		/*
 		 * Don't fault in device private pages owned by the caller,
 		 * just report the PFN.
@@ -416,6 +438,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 
 	new_pfn_flags = pte_pfn(pte) | cpu_flags;
 out:
+	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
+						addr + PAGE_SIZE,
+						hmm_pfn);
 	*hmm_pfn = (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags;
 	return 0;
 
@@ -427,7 +452,19 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 	} else {
 		pte_unmap(ptep);
 	}
-	/* Fault any virtual address we were asked to fault */
+	/*
+	 * This pte needs a fault, so it is not our migration entry: a pfn
+	 * collected here earlier is gone, roll it back before faulting.
+	 */
+	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
+						addr + PAGE_SIZE, hmm_pfn);
+	/*
+	 * Fault any virtual address we were asked to fault. When migrating,
+	 * only this page: later ptes may be our own migration entries or
+	 * caller owned device private pages, which must not be faulted.
+	 */
+	if (hmm_select_migrate(range))
+		end = addr + PAGE_SIZE;
 	return hmm_record_fault(addr, end, required_fault, walk);
 }
 
@@ -454,6 +491,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 		if (softleaf_is_device_private_write(entry))
 			cpu_flags |= HMM_PFN_WRITE;
 
+		hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+							start, end,
+							hmm_pfns);
 		/*
 		 * Fully populate the PFN list though subsequent PFNs could be
 		 * inferred, because drivers which are not yet aware of large
@@ -475,6 +515,10 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
 				spin_unlock(hmm_vma_walk->ptl);
 				hmm_vma_walk->pmdlocked = false;
 			}
+			/* Entries collected from a vanished page table are gone */
+			hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+								addr, end,
+								hmm_pfns);
 			return hmm_record_fault(addr, end, required_fault, walk);
 		}
 		else
@@ -561,6 +605,51 @@ static int migrate_vma_split_folio(struct folio *folio,
 	return __migrate_vma_split_folio(folio, fault_page);
 }
 
+/*
+ * Due to dropping ptl locks for splitting for instance, would we
+ * overwrite already collected pfns? This could happen when pmd
+ * pointing to a page table has vanished and been replaced with
+ * a leaf pmd, or another page table. In that case unref and unlock
+ * the folios, the pfns of which were collected from the disappeared
+ * page tables.
+ *
+ * Migrate mode drives fault/migration requests only via range->default_flags.
+ * (HMM_PFN_VALID | HMM_PFN_MIGRATE) denotes a collected entry and can safely
+ * be cleared from hmm_pfn[].
+ */
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+						    unsigned long start,
+						    unsigned long end,
+						    unsigned long *hmm_pfn)
+{
+	struct hmm_range *range = hmm_vma_walk->range;
+	struct migrate_vma *migrate = range->migrate;
+	struct folio *fault_folio = NULL;
+	enum migrate_vma_info minfo;
+	struct folio *folio;
+	unsigned long i;
+
+	minfo = hmm_select_migrate(range);
+	if (!minfo)
+		return;
+
+	WARN_ON_ONCE(!migrate);
+
+	fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
+
+	for (i = 0; start < end; start += PAGE_SIZE, i++) {
+		if (hmm_pfn_collected(hmm_pfn[i])) {
+			folio = page_folio(hmm_pfn_to_page(hmm_pfn[i]));
+			if (folio != fault_folio)
+				folio_unlock(folio);
+			folio_put(folio);
+			hmm_pfn[i] = hmm_pfn_rollback_collected(hmm_pfn[i]);
+
+		}
+	}
+}
+
 static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
 					      pmd_t *pmdp,
 					      unsigned long start,
@@ -731,6 +820,11 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
 	pte = ptep_get(ptep);
 
 	if (pte_none(pte)) {
+		hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+							addr,
+							addr + PAGE_SIZE,
+							hmm_pfn);
+
 		if (vma_is_anonymous(walk->vma)) {
 			*hmm_pfn &= HMM_PFN_INOUT_FLAGS;
 			*hmm_pfn |= HMM_PFN_MIGRATE;
@@ -783,6 +877,10 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
 		pfn = pte_pfn(pte);
 		if (is_zero_pfn(pfn) &&
 		    (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+			hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+								addr,
+								addr + PAGE_SIZE,
+								hmm_pfn);
 			/* Like pte_none(): only anonymous memory can be populated */
 			if (!vma_is_anonymous(walk->vma))
 				goto out;
@@ -929,6 +1027,13 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
 }
 
 #else
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+						    unsigned long start,
+						    unsigned long end,
+						    unsigned long *hmm_pfn)
+{
+}
+
 static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
 					      pmd_t *pmdp,
 					      unsigned long start,
@@ -1068,6 +1173,16 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 			hmm_vma_walk->pmdlocked = false;
 		}
 
+		/* Our own migration entry, collected earlier in this walk */
+		if (minfo && hmm_pfn_collected(hmm_pfns_start[0]) &&
+		    hmm_pfn_to_page(hmm_pfns_start[0]) ==
+		    softleaf_to_page(softleaf_from_pmd(pmd)))
+			return 0;
+
+		/* Not ours: entries collected from a vanished page table are gone */
+		hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, start, end,
+							hmm_pfns_start);
+
 		if (required_fault) {
 			/*
 			 * Someone else's migration: when migrating, wait in
@@ -1171,6 +1286,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 		ptep = pte_offset_map(pmdp, addr);
 	}
 	if (!ptep) {
+		/*
+		 * The pte table is gone: re-walk from start, entries collected
+		 * from the old table are rolled back as they are revisited.
+		 */
 		addr = start;
 		goto again;
 	}
@@ -1192,6 +1311,12 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
 				flush_tlb_range(walk->vma, start, addr);
 				unmapped = false;
 			}
+			/*
+			 * The pte lock was dropped and the pte table may have
+			 * changed: re-walk from start. Our own migration entries
+			 * are kept, anything else collected is rolled back.
+			 */
+			addr = start;
 			goto again;
 		}
 		if (r) {
-- 
2.55.0

[PATCH v17 09/12] mm: enable device page migration from HMM pagewalk

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

HMM pagewalk has now the machinery to do the first phase of
device page migration, collecting the pfns and installing
migration ptes.

Enable migration in hmm_range_fault(), and change migrate_vma_setup()
to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and
MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(),
to request for faulting missing pages, and requesting write
access.

Also, the migrate_vma_collect*() based paths are now unused,
so delete them.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 include/linux/migrate.h |  10 +-
 mm/hmm.c                |  33 ++-
 mm/migrate_device.c     | 563 ++++------------------------------------
 3 files changed, 84 insertions(+), 522 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index dd7dd7ec6995..3780c4c116b8 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -181,6 +181,8 @@ enum migrate_vma_info {
 	MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
 	MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
 	MIGRATE_VMA_SELECT_COMPOUND = 1 << 3,
+	MIGRATE_VMA_FAULT = 1 << 4,
+	MIGRATE_VMA_WRITE = 1 << 5,
 };
 
 struct migrate_vma {
@@ -218,10 +220,14 @@ struct migrate_vma {
 	struct page		*fault_page;
 };
 
-// TODO: enable migration
 static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
 {
-	return 0;
+	enum migrate_vma_info minfo;
+
+	minfo = (range->default_flags & HMM_PFN_REQ_MIGRATE) ?
+		range->migrate->flags : 0;
+
+	return minfo;
 }
 
 #ifdef CONFIG_MMU_NOTIFIER
diff --git a/mm/hmm.c b/mm/hmm.c
index 1c8a3fa29c4e..0b38bf84c369 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -1658,14 +1658,43 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
  *		the invalidation to finish.
  * -EFAULT:     A page was requested to be valid and could not be made valid
  *              ie it has no backing VMA or it is illegal to access
+ * -ERANGE:     The range crosses multiple VMAs, or space for hmm_pfns array
+ *              is too low.
  *
  * This is similar to get_user_pages(), except that it can read the page tables
  * without mutating them (ie causing faults).
  *
  * The mmap lock must be held by the caller and will remain held on return.
  * New users should prefer hmm_range_fault_unlocked_timeout() unless they
- * specifically need to keep the mmap lock held across the call. This helper
- * cannot support VMAs whose fault handlers need to drop the mmap lock.
+ * specifically need to keep the mmap lock held across the call like while
+ * migrating. This helper cannot support VMAs whose fault handlers need to
+ * drop the mmap lock.
+ *
+ * If want to do migration after faulting, call hmm_range_fault() with
+ * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally
+ * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate field.
+ * range->migrate->vma will be populated during the call,
+ * and must be stable across the whole migrate process, which is
+ * why mmap_lock must be held around this call.
+ *
+ * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on
+ * return and the caller takes responsibility for completing or aborting it.
+ *
+ * On success, the caller must call migrate_hmm_range_setup() and may then
+ * proceed with the normal migrate_vma sequence: prepare destination pages,
+ * call migrate_vma_pages(), update device mappings as needed, and finally
+ * call migrate_vma_finalize().
+ *
+ * On -EBUSY, the caller may retry hmm_range_fault() using the same range and
+ * PFN array without undoing entries collected by the previous attempt. If
+ * the caller stops retrying, it must abort the partial migration.
+ *
+ * On any other error, or when abandoning a retry, the caller must call
+ * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination
+ * entries, and migrate_vma_finalize() to abort the partial migration.
+ *
+ * The mmap read lock must remain held until the migration has either been
+ * completed or aborted.
  */
 int hmm_range_fault(struct hmm_range *range)
 {
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index ba6775a0d37e..266f9c6d7b60 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -18,519 +18,6 @@
 #include <asm/tlbflush.h>
 #include "internal.h"
 
-static int migrate_vma_collect_skip(unsigned long start,
-				    unsigned long end,
-				    struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	unsigned long addr;
-
-	for (addr = start; addr < end; addr += PAGE_SIZE) {
-		migrate->dst[migrate->npages] = 0;
-		migrate->src[migrate->npages++] = 0;
-	}
-
-	return 0;
-}
-
-static int migrate_vma_collect_hole(unsigned long start,
-				    unsigned long end,
-				    __always_unused int depth,
-				    struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	unsigned long addr;
-
-	/* Only allow populating anonymous memory. */
-	if (!vma_is_anonymous(walk->vma))
-		return migrate_vma_collect_skip(start, end, walk);
-
-	if (thp_migration_supported() &&
-		(migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
-		(IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
-		 IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
-		migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE |
-						MIGRATE_PFN_COMPOUND;
-		migrate->dst[migrate->npages] = 0;
-		migrate->npages++;
-		migrate->cpages++;
-
-		/*
-		 * Collect the remaining entries as holes, in case we
-		 * need to split later
-		 */
-		return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
-	}
-
-	for (addr = start; addr < end; addr += PAGE_SIZE) {
-		migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE;
-		migrate->dst[migrate->npages] = 0;
-		migrate->npages++;
-		migrate->cpages++;
-	}
-
-	return 0;
-}
-
-/**
- * migrate_vma_split_folio() - Helper function to split a THP folio
- * @folio: the folio to split
- * @fault_page: struct page associated with the fault if any
- *
- * If @folio is not the folio containing @fault_page, the caller must hold a
- * reference on @folio. The helper consumes that reference.
- *
- * Returns 0 on success
- */
-static int migrate_vma_split_folio(struct folio *folio,
-				   struct page *fault_page)
-{
-	int ret;
-	struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
-	struct folio *new_fault_folio = NULL;
-
-	if (folio != fault_folio)
-		folio_lock(folio);
-
-	ret = split_folio(folio);
-	if (ret) {
-		if (folio != fault_folio) {
-			folio_unlock(folio);
-			folio_put(folio);
-		}
-		return ret;
-	}
-
-	new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
-
-	/*
-	 * Ensure the lock is held on the correct
-	 * folio after the split
-	 */
-	if (!new_fault_folio) {
-		folio_unlock(folio);
-		folio_put(folio);
-	} else if (folio != new_fault_folio) {
-		if (new_fault_folio != fault_folio) {
-			folio_get(new_fault_folio);
-			folio_lock(new_fault_folio);
-		}
-		folio_unlock(folio);
-		folio_put(folio);
-	}
-
-	return 0;
-}
-
-/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the
- * folio for device private pages.
- * @pmdp: pointer to pmd entry
- * @start: start address of the range for migration
- * @end: end address of the range for migration
- * @walk: mm_walk callback structure
- * @fault_folio: folio associated with the fault if any
- *
- * Collect the huge pmd entry at @pmdp for migration and set the
- * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that
- * migration will occur at HPAGE_PMD granularity
- */
-static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start,
-					unsigned long end, struct mm_walk *walk,
-					struct folio *fault_folio)
-{
-	struct mm_struct *mm = walk->mm;
-	struct folio *folio;
-	struct migrate_vma *migrate = walk->private;
-	spinlock_t *ptl;
-	int ret;
-	unsigned long write = 0;
-
-	ptl = pmd_lock(mm, pmdp);
-	if (pmd_none(*pmdp)) {
-		spin_unlock(ptl);
-		return migrate_vma_collect_hole(start, end, -1, walk);
-	}
-
-	if (pmd_trans_huge(*pmdp)) {
-		if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_skip(start, end, walk);
-		}
-
-		folio = pmd_folio(*pmdp);
-		if (is_huge_zero_folio(folio)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_hole(start, end, -1, walk);
-		}
-		if (pmd_write(*pmdp))
-			write = MIGRATE_PFN_WRITE;
-	} else if (!pmd_present(*pmdp)) {
-		const softleaf_t entry = softleaf_from_pmd(*pmdp);
-
-		if (!softleaf_is_device_private(entry) ||
-		    !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_skip(start, end, walk);
-		}
-
-		folio = softleaf_to_folio(entry);
-		if (folio->pgmap->owner != migrate->pgmap_owner) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_skip(start, end, walk);
-		}
-
-		if (softleaf_is_device_private_write(entry))
-			write = MIGRATE_PFN_WRITE;
-	} else {
-		spin_unlock(ptl);
-		return -EAGAIN;
-	}
-
-	folio_get(folio);
-	if (folio != fault_folio && unlikely(!folio_trylock(folio))) {
-		spin_unlock(ptl);
-		folio_put(folio);
-		return migrate_vma_collect_skip(start, end, walk);
-	}
-
-	if (thp_migration_supported() &&
-		(migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
-		(IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
-		 IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
-
-		struct page_vma_mapped_walk pvmw = {
-			.ptl = ptl,
-			.address = start,
-			.pmd = pmdp,
-			.vma = walk->vma,
-		};
-
-		unsigned long pfn = page_to_pfn(folio_page(folio, 0));
-
-		migrate->src[migrate->npages] = migrate_pfn(pfn) | write
-						| MIGRATE_PFN_MIGRATE
-						| MIGRATE_PFN_COMPOUND;
-		migrate->dst[migrate->npages++] = 0;
-		migrate->cpages++;
-		ret = set_pmd_migration_entry(&pvmw, folio_page(folio, 0));
-		if (ret) {
-			migrate->npages--;
-			migrate->cpages--;
-			migrate->src[migrate->npages] = 0;
-			migrate->dst[migrate->npages] = 0;
-			goto fallback;
-		}
-		migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
-		spin_unlock(ptl);
-		return 0;
-	}
-
-fallback:
-	spin_unlock(ptl);
-	if (!folio_test_large(folio))
-		goto done;
-	ret = split_folio(folio);
-	if (fault_folio != folio)
-		folio_unlock(folio);
-	folio_put(folio);
-	if (ret)
-		return migrate_vma_collect_skip(start, end, walk);
-	if (pmd_none(pmdp_get_lockless(pmdp)))
-		return migrate_vma_collect_hole(start, end, -1, walk);
-
-done:
-	return -ENOENT;
-}
-
-static int migrate_vma_collect_pmd(pmd_t *pmdp,
-				   unsigned long start,
-				   unsigned long end,
-				   struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	struct vm_area_struct *vma = walk->vma;
-	struct mm_struct *mm = vma->vm_mm;
-	unsigned long addr = start, unmapped = 0;
-	spinlock_t *ptl;
-	struct folio *fault_folio = migrate->fault_page ?
-		page_folio(migrate->fault_page) : NULL;
-	pte_t *ptep;
-
-again:
-	if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) {
-		int ret = migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_folio);
-
-		if (ret == -EAGAIN)
-			goto again;
-		if (ret == 0)
-			return 0;
-	}
-
-	ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
-	if (!ptep)
-		goto again;
-	lazy_mmu_mode_enable();
-	ptep += (addr - start) / PAGE_SIZE;
-
-	for (; addr < end; addr += PAGE_SIZE, ptep++) {
-		struct dev_pagemap *pgmap;
-		unsigned long mpfn = 0, pfn;
-		struct folio *folio;
-		struct page *page;
-		softleaf_t entry;
-		pte_t pte;
-
-		pte = ptep_get(ptep);
-
-		if (pte_none(pte)) {
-			if (vma_is_anonymous(vma)) {
-				mpfn = MIGRATE_PFN_MIGRATE;
-				migrate->cpages++;
-			}
-			goto next;
-		}
-
-		if (!pte_present(pte)) {
-			/*
-			 * Only care about unaddressable device page special
-			 * page table entry. Other special swap entries are not
-			 * migratable, and we ignore regular swapped page.
-			 */
-			entry = softleaf_from_pte(pte);
-			if (!softleaf_is_device_private(entry))
-				goto next;
-
-			page = softleaf_to_page(entry);
-			pgmap = page_pgmap(page);
-			if (!(migrate->flags &
-				MIGRATE_VMA_SELECT_DEVICE_PRIVATE) ||
-			    pgmap->owner != migrate->pgmap_owner)
-				goto next;
-
-			folio = page_folio(page);
-			if (folio_test_large(folio)) {
-				int ret;
-
-				/* migrate_vma_split_folio() consumes this reference */
-				if (folio != fault_folio)
-					folio_get(folio);
-				lazy_mmu_mode_disable();
-				pte_unmap_unlock(ptep, ptl);
-				ret = migrate_vma_split_folio(folio,
-							  migrate->fault_page);
-
-				if (ret) {
-					if (unmapped)
-						flush_tlb_range(walk->vma, start, end);
-
-					return migrate_vma_collect_skip(addr, end, walk);
-				}
-
-				goto again;
-			}
-
-			mpfn = migrate_pfn(page_to_pfn(page)) |
-					MIGRATE_PFN_MIGRATE;
-			if (softleaf_is_device_private_write(entry))
-				mpfn |= MIGRATE_PFN_WRITE;
-		} else {
-			pfn = pte_pfn(pte);
-			if (is_zero_pfn(pfn) &&
-			    (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-				mpfn = MIGRATE_PFN_MIGRATE;
-				migrate->cpages++;
-				goto next;
-			}
-			page = vm_normal_page(migrate->vma, addr, pte);
-			if (page && !is_zone_device_page(page) &&
-			    !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-				goto next;
-			} else if (page && is_device_coherent_page(page)) {
-				pgmap = page_pgmap(page);
-
-				if (!(migrate->flags &
-					MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
-					pgmap->owner != migrate->pgmap_owner)
-					goto next;
-			}
-			folio = page ? page_folio(page) : NULL;
-			if (folio && folio_test_large(folio)) {
-				int ret;
-
-				/* migrate_vma_split_folio() consumes this reference */
-				if (folio != fault_folio)
-					folio_get(folio);
-				lazy_mmu_mode_disable();
-				pte_unmap_unlock(ptep, ptl);
-				ret = migrate_vma_split_folio(folio,
-							  migrate->fault_page);
-
-				if (ret) {
-					if (unmapped)
-						flush_tlb_range(walk->vma, start, end);
-
-					return migrate_vma_collect_skip(addr, end, walk);
-				}
-
-				goto again;
-			}
-			mpfn = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
-			mpfn |= pte_write(pte) ? MIGRATE_PFN_WRITE : 0;
-		}
-
-		if (!page || !page->mapping) {
-			mpfn = 0;
-			goto next;
-		}
-
-		/*
-		 * By getting a reference on the folio we pin it and that blocks
-		 * any kind of migration. Side effect is that it "freezes" the
-		 * pte.
-		 *
-		 * We drop this reference after isolating the folio from the lru
-		 * for non device folio (device folio are not on the lru and thus
-		 * can't be dropped from it).
-		 */
-		folio = page_folio(page);
-		folio_get(folio);
-
-		/*
-		 * We rely on folio_trylock() to avoid deadlock between
-		 * concurrent migrations where each is waiting on the others
-		 * folio lock. If we can't immediately lock the folio we fail this
-		 * migration as it is only best effort anyway.
-		 *
-		 * If we can lock the folio it's safe to set up a migration entry
-		 * now. In the common case where the folio is mapped once in a
-		 * single process setting up the migration entry now is an
-		 * optimisation to avoid walking the rmap later with
-		 * try_to_migrate().
-		 */
-		if (fault_folio == folio || folio_trylock(folio)) {
-			bool anon_exclusive;
-			pte_t swp_pte;
-
-			if (pte_present(pte))
-				flush_cache_page(vma, addr, pte_pfn(pte));
-			anon_exclusive = folio_test_anon(folio) &&
-					  PageAnonExclusive(page);
-			if (anon_exclusive) {
-				pte = ptep_clear_flush(vma, addr, ptep);
-
-				if (folio_try_share_anon_rmap_pte(folio, page)) {
-					set_pte_at(mm, addr, ptep, pte);
-					if (fault_folio != folio)
-						folio_unlock(folio);
-					folio_put(folio);
-					mpfn = 0;
-					goto next;
-				}
-			} else {
-				pte = ptep_get_and_clear(mm, addr, ptep);
-			}
-
-			migrate->cpages++;
-
-			/* Set the dirty flag on the folio now the pte is gone. */
-			if (pte_present(pte) && pte_dirty(pte))
-				folio_mark_dirty(folio);
-
-			/* Setup special migration page table entry */
-			if (mpfn & MIGRATE_PFN_WRITE)
-				entry = make_writable_migration_entry(
-							page_to_pfn(page));
-			else if (anon_exclusive)
-				entry = make_readable_exclusive_migration_entry(
-							page_to_pfn(page));
-			else
-				entry = make_readable_migration_entry(
-							page_to_pfn(page));
-			if (pte_present(pte)) {
-				if (pte_young(pte))
-					entry = make_migration_entry_young(entry);
-				if (pte_dirty(pte))
-					entry = make_migration_entry_dirty(entry);
-			}
-			swp_pte = swp_entry_to_pte(entry);
-			if (pte_present(pte)) {
-				if (pte_soft_dirty(pte))
-					swp_pte = pte_swp_mksoft_dirty(swp_pte);
-				if (pte_uffd(pte))
-					swp_pte = pte_swp_mkuffd(swp_pte);
-			} else {
-				if (pte_swp_soft_dirty(pte))
-					swp_pte = pte_swp_mksoft_dirty(swp_pte);
-				if (pte_swp_uffd(pte))
-					swp_pte = pte_swp_mkuffd(swp_pte);
-			}
-			set_pte_at(mm, addr, ptep, swp_pte);
-
-			/*
-			 * This is like regular unmap: we remove the rmap and
-			 * drop the folio refcount. The folio won't be freed, as
-			 * we took a reference just above.
-			 */
-			folio_remove_rmap_pte(folio, page, vma);
-			folio_put(folio);
-
-			if (pte_present(pte))
-				unmapped++;
-		} else {
-			folio_put(folio);
-			mpfn = 0;
-		}
-
-next:
-		migrate->dst[migrate->npages] = 0;
-		migrate->src[migrate->npages++] = mpfn;
-	}
-
-	/* Only flush the TLB if we actually modified any entries */
-	if (unmapped)
-		flush_tlb_range(walk->vma, start, end);
-
-	lazy_mmu_mode_disable();
-	pte_unmap_unlock(ptep - 1, ptl);
-
-	return 0;
-}
-
-static const struct mm_walk_ops migrate_vma_walk_ops = {
-	.pmd_entry		= migrate_vma_collect_pmd,
-	.pte_hole		= migrate_vma_collect_hole,
-	.walk_lock		= PGWALK_RDLOCK,
-};
-
-/*
- * migrate_vma_collect() - collect pages over a range of virtual addresses
- * @migrate: migrate struct containing all migration information
- *
- * This will walk the CPU page table. For each virtual address backed by a
- * valid page, it updates the src array and takes a reference on the page, in
- * order to pin the page until we lock it and unmap it.
- */
-static void migrate_vma_collect(struct migrate_vma *migrate)
-{
-	struct mmu_notifier_range range;
-
-	/*
-	 * Note that the pgmap_owner is passed to the mmu notifier callback so
-	 * that the registered device driver can skip invalidating device
-	 * private page mappings that won't be migrated.
-	 */
-	mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
-		migrate->vma->vm_mm, migrate->start, migrate->end,
-		migrate->pgmap_owner);
-	mmu_notifier_invalidate_range_start(&range);
-
-	walk_page_range_vma(migrate->vma, migrate->start, migrate->end,
-			&migrate_vma_walk_ops, migrate);
-
-	mmu_notifier_invalidate_range_end(&range);
-	migrate->end = migrate->start + (migrate->npages << PAGE_SHIFT);
-}
-
 /*
  * migrate_vma_check_page() - check if page is pinned or not
  * @page: struct page to check
@@ -677,8 +164,9 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
  * migrate_vma_setup() - prepare to migrate a range of memory
  * @args: contains the vma, start, and pfns arrays for the migration
  *
- * Returns: negative errno on failures, 0 when 0 or more pages were migrated
- * without an error.
+ * Returns: -EINVAL on invalid arguments, 0 otherwise. A 0 return does not
+ * imply anything was collected; on return @args->cpages holds the number of
+ * pages prepared for migration (possibly 0). See the note on faulting below.
  *
  * Prepare to migrate a range of memory virtual address range by collecting all
  * the pages backing each virtual address in the range, saving them inside the
@@ -688,6 +176,20 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
  * corresponding src array entry.  Then restores any pages that are pinned, by
  * remapping and unlocking those pages.
  *
+ * By default migrate_vma_setup() is best-effort: virtual addresses that cannot
+ * be collected are left with an empty src[] entry and the rest of the range
+ * still proceeds.
+ *
+ * The optional flags MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE in @args->flags
+ * additionally fault in missing pages before collecting them, for read and for
+ * write respectively. Unlike the best-effort default, if such a fault cannot be
+ * satisfied the whole range is aborted: the partial collection is rolled back,
+ * @args->src is cleared, @args->cpages and @args->npages are reset to 0,
+ * and migrate_vma_setup() still returns 0. The fault error is therefore not
+ * observable through this interface. Callers that need to act on fault failures
+ * should instead drive the migration directly via hmm_range_fault() followed by
+ * migrate_hmm_range_setup(), which preserves the error code.
+ *
  * The caller should then allocate destination memory and copy source memory to
  * it for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATE
  * flag set).  Once these are allocated and copied, the caller must update each
@@ -741,9 +243,20 @@ int migrate_vma_setup(struct migrate_vma *args)
 {
 	const struct vm_area_struct *vma = args->vma;
 	long nr_pages = (args->end - args->start) >> PAGE_SHIFT;
+	struct hmm_range range = {
+		.notifier = NULL,
+		.hmm_pfns = args->src,
+		.dev_private_owner = args->pgmap_owner,
+		.migrate = args,
+		.default_flags = HMM_PFN_REQ_MIGRATE
+	};
+	int ret;
 
 	args->start &= PAGE_MASK;
 	args->end &= PAGE_MASK;
+	range.start = args->start;
+	range.end = args->end;
+
 	if (!vma)
 		return -EINVAL;
 	/* The migrate_vma page walk has no hugetlb support. */
@@ -766,10 +279,24 @@ int migrate_vma_setup(struct migrate_vma *args)
 	args->cpages = 0;
 	args->npages = 0;
 
-	migrate_vma_collect(args);
+	if (args->flags & MIGRATE_VMA_FAULT)
+		range.default_flags |= HMM_PFN_REQ_FAULT;
 
-	if (args->cpages)
-		migrate_vma_unmap(args);
+	if (args->flags & MIGRATE_VMA_WRITE)
+		range.default_flags |= HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE;
+
+	ret = hmm_range_fault(&range);
+
+	migrate_hmm_range_setup(&range);
+
+	/* Remove migration PTEs */
+	if (ret) {
+		migrate_vma_pages(args);
+		migrate_vma_finalize(args);
+		memset(args->src, 0, sizeof(*args->src) * nr_pages);
+		args->cpages = 0;
+		args->npages = 0;
+	}
 
 	/*
 	 * At this point pages are locked and unmapped, and thus they have
-- 
2.55.0

[PATCH v17 10/12] lib/test_hmm: add a new testcase for the migrate on fault

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Enhance the hmm test driver (lib/test_hmm) with a migrate on fault
case: a single hmm_range_fault() walk with HMM_PFN_REQ_FAULT |
HMM_PFN_REQ_MIGRATE faults in and collects the range, which is then
migrated to device private memory and mapped into the dmirror page
table. The selftest leaves the first two pages unwritten, so page 0
has to be faulted in and page 1 maps the zero page.

If mmu_interval_read_retry() trips after migrate_vma_pages(), the
dmirror mapping is skipped and the -EBUSY retry re-walks. Normally the
trip comes from an unmap or zap: the migration ptes are gone,
migrate_vma_finalize() releases the destination pages and the retry
migrates again. Pages committed to device memory without a dmirror
mapping (e.g. after a huge zero pmd split in migrate_vma_insert_page())
are still consistent, since any invalidation also clears the dmirror
page table, and a real device would map them on its next fault. The
selftest doesn't hit that case: the unwritten pages share a pmd with
written ones, so there is no huge zero pmd.

Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Marco Pagani <marpagan@redhat.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 lib/test_hmm.c                         | 172 +++++++++++++++++++++++--
 lib/test_hmm_uapi.h                    |  21 +--
 tools/testing/selftests/mm/hmm-tests.c |  56 ++++++++
 3 files changed, 226 insertions(+), 23 deletions(-)

diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index f05f801b8404..2045896493ae 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -36,6 +36,7 @@
 #define DMIRROR_RANGE_FAULT_TIMEOUT	1000
 #define DEVMEM_CHUNK_SIZE		(256 * 1024 * 1024U)
 #define DEVMEM_CHUNKS_RESERVE		16
+#define PFNS_ARRAY_SIZE			64
 
 /*
  * For device_private pages, dpage is just a dummy struct page
@@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror,
 	struct mm_struct *mm = dmirror->notifier.mm;
 	unsigned long timeout =
 		jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+	bool migrate = range->default_flags & HMM_PFN_REQ_MIGRATE;
 	int ret;
 
 	while (true) {
@@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror,
 		}
 
 		range->notifier_seq = mmu_interval_read_begin(range->notifier);
-		mmap_read_lock(mm);
-		ret = hmm_range_fault(range);
-		mmap_read_unlock(mm);
+
+		/* mmap lock held for whole migrate on fault operation */
+		if (!migrate) {
+			mmap_read_lock(mm);
+			ret = hmm_range_fault(range);
+			mmap_read_unlock(mm);
+		} else {
+			ret = hmm_range_fault(range);
+		}
 		if (ret) {
 			if (ret == -EBUSY)
 				continue;
@@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror,
 		break;
 	}
 
-	ret = dmirror_do_fault(dmirror, range);
+	/* update device page table after migration */
+	if (!migrate)
+		ret = dmirror_do_fault(dmirror, range);
 
 	mutex_unlock(&dmirror->mutex);
 out:
@@ -994,8 +1004,12 @@ static int dmirror_atomic_map(unsigned long addr, struct page *page,
 	return 0;
 }
 
-static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
-					    struct dmirror *dmirror)
+/*
+ * Map the migrated pages into the device's page table. Caller must hold
+ * dmirror->mutex.
+ */
+static int __dmirror_migrate_map_locked(struct migrate_vma *args,
+					struct dmirror *dmirror)
 {
 	unsigned long start = args->start;
 	unsigned long end = args->end;
@@ -1005,9 +1019,6 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
 	const unsigned long start_pfn = start >> PAGE_SHIFT;
 	const unsigned long end_pfn = end >> PAGE_SHIFT;
 
-	/* Map the migrated pages into the device's page tables. */
-	mutex_lock(&dmirror->mutex);
-
 	for (pfn = start_pfn; pfn < end_pfn; pfn++, src++, dst++) {
 		struct page *dpage;
 		void *entry;
@@ -1036,17 +1047,27 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
 			if (*dst & MIGRATE_PFN_WRITE)
 				entry = xa_tag_pointer(entry, DPT_XA_TAG_WRITE);
 			entry = xa_store(&dmirror->pt, pfn + i, entry, GFP_ATOMIC);
-			if (xa_is_err(entry)) {
-				mutex_unlock(&dmirror->mutex);
+			if (xa_is_err(entry))
 				return xa_err(entry);
-			}
 		}
 	}
 
-	mutex_unlock(&dmirror->mutex);
 	return 0;
 }
 
+static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
+					    struct dmirror *dmirror)
+{
+	int ret;
+
+	/* Map the migrated pages into the device's page tables. */
+	mutex_lock(&dmirror->mutex);
+	ret = __dmirror_migrate_map_locked(args, dmirror);
+	mutex_unlock(&dmirror->mutex);
+
+	return ret;
+}
+
 static int dmirror_exclusive(struct dmirror *dmirror,
 			     struct hmm_dmirror_cmd *cmd)
 {
@@ -1377,6 +1398,127 @@ static int dmirror_migrate_to_device(struct dmirror *dmirror,
 	return ret;
 }
 
+static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range *range)
+{
+	struct migrate_vma *migrate = range->migrate;
+	unsigned long timeout =
+		jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+	int ret;
+
+	do {
+		mmap_read_lock(dmirror->notifier.mm);
+
+		/* Fault-in pages for migration */
+		ret = dmirror_range_fault(dmirror, range);
+		/*
+		 * Set this up even on error: hmm_range_fault() may have collected
+		 * part of the range before failing.
+		 */
+		migrate_hmm_range_setup(range);
+		if (ret) {
+			/*
+			 * dst[] entries are empty, so this marks every collected
+			 * migration as failed. finalize then restores the source
+			 * mappings and drops the associated locks/references.
+			 */
+			migrate_vma_pages(migrate);
+			migrate_vma_finalize(migrate);
+			goto unlock;
+		}
+
+		pr_debug("Migrating from sys mem to device mem\n");
+
+		dmirror_migrate_alloc_and_copy(migrate, dmirror);
+		migrate_vma_pages(migrate);
+
+		mutex_lock(&dmirror->mutex);
+		if (mmu_interval_read_retry(&dmirror->notifier,
+					    range->notifier_seq))
+			ret = -EBUSY;
+		else
+			ret = __dmirror_migrate_map_locked(migrate, dmirror);
+		mutex_unlock(&dmirror->mutex);
+
+		migrate_vma_finalize(migrate);
+unlock:
+		mmap_read_unlock(dmirror->notifier.mm);
+	} while (ret == -EBUSY && !time_after(jiffies, timeout));
+
+	return ret;
+}
+
+static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror,
+					       struct hmm_dmirror_cmd *cmd)
+{
+	unsigned long start, size, end, next;
+	unsigned long src_pfns[PFNS_ARRAY_SIZE] = { 0 };
+	unsigned long dst_pfns[PFNS_ARRAY_SIZE] = { 0 };
+	struct migrate_vma migrate = { 0 };
+	struct hmm_range range = { 0 };
+	struct dmirror_bounce bounce;
+	int ret = 0;
+
+	/* Whole range */
+	start = cmd->addr;
+	size = cmd->npages << PAGE_SHIFT;
+	end = start + size;
+
+	if (!mmget_not_zero(dmirror->notifier.mm)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	migrate.pgmap_owner = dmirror->mdevice;
+	migrate.src = src_pfns;
+	migrate.dst = dst_pfns;
+	migrate.flags = MIGRATE_VMA_SELECT_SYSTEM;
+
+	range.migrate = &migrate;
+	range.hmm_pfns = src_pfns;
+	range.pfn_flags_mask = 0;
+	range.default_flags = HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE;
+	range.dev_private_owner = dmirror->mdevice;
+	range.notifier = &dmirror->notifier;
+
+	for (next = start; next < end; next = range.end) {
+		range.start = next;
+		range.end = min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT));
+
+		pr_debug("Fault and migrate range start:%#lx end:%#lx\n",
+			 range.start, range.end);
+
+		ret = do_fault_and_migrate(dmirror, &range);
+		if (ret)
+			goto out_mmput;
+	}
+
+	/*
+	 * Return the migrated data for verification.
+	 * Only for pages in device zone
+	 */
+	ret = dmirror_bounce_init(&bounce, start, size);
+	if (ret)
+		goto out_mmput;
+
+	mutex_lock(&dmirror->mutex);
+	ret = dmirror_do_read(dmirror, start, end, &bounce);
+	mutex_unlock(&dmirror->mutex);
+	if (ret == 0) {
+		ret = copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size);
+		if (ret)
+			ret = -EFAULT;
+	}
+
+	cmd->cpages = bounce.cpages;
+	dmirror_bounce_fini(&bounce);
+
+
+out_mmput:
+	mmput(dmirror->notifier.mm);
+out:
+	return ret;
+}
+
 static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *range,
 			    unsigned char *perm, unsigned long entry)
 {
@@ -1645,6 +1787,10 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
 		ret = dmirror_migrate_to_device(dmirror, &cmd);
 		break;
 
+	case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV:
+		ret = dmirror_fault_and_migrate_to_device(dmirror, &cmd);
+		break;
+
 	case HMM_DMIRROR_MIGRATE_TO_SYS:
 		ret = dmirror_migrate_to_system(dmirror, &cmd);
 		break;
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index ea9b0ec404fb..67d267a3b999 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -29,16 +29,17 @@ struct hmm_dmirror_cmd {
 };
 
 /* Expose the address space of the calling process through hmm device file */
-#define HMM_DMIRROR_READ		_IOWR('H', 0x00, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_WRITE		_IOWR('H', 0x01, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_DEV	_IOWR('H', 0x02, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_SYS	_IOWR('H', 0x03, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_SNAPSHOT		_IOWR('H', 0x04, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_EXCLUSIVE		_IOWR('H', 0x05, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_CHECK_EXCLUSIVE	_IOWR('H', 0x06, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_RELEASE		_IOWR('H', 0x07, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_FLAGS		_IOWR('H', 0x08, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_READ_UNLOCKED	_IOWR('H', 0x09, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ			_IOWR('H', 0x00, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_WRITE			_IOWR('H', 0x01, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_DEV		_IOWR('H', 0x02, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_SYS		_IOWR('H', 0x03, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_SNAPSHOT			_IOWR('H', 0x04, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_EXCLUSIVE			_IOWR('H', 0x05, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_CHECK_EXCLUSIVE		_IOWR('H', 0x06, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_RELEASE			_IOWR('H', 0x07, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_FLAGS			_IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED		_IOWR('H', 0x09, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV	_IOWR('H', 0x0a, struct hmm_dmirror_cmd)
 
 #define HMM_DMIRROR_FLAG_FAIL_ALLOC	(1ULL << 0)
 
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index e5f273ca84c1..bacf573d83a9 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -333,6 +333,13 @@ static int hmm_migrate_sys_to_dev(int fd,
 	return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages);
 }
 
+static int hmm_migrate_on_fault_sys_to_dev(int fd,
+					   struct hmm_buffer *buffer,
+					   unsigned long npages)
+{
+	return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, npages);
+}
+
 static int hmm_migrate_dev_to_sys(int fd,
 				   struct hmm_buffer *buffer,
 				   unsigned long npages)
@@ -937,6 +944,55 @@ TEST_F(hmm, migrate)
 	hmm_buffer_free(buffer);
 }
 
+
+/*
+ * Fault and migrate anonymous memory to device private memory.
+ */
+TEST_F(hmm, migrate_on_fault)
+{
+	struct hmm_buffer *buffer;
+	unsigned long npages;
+	unsigned long size;
+	unsigned long i;
+	int *ptr;
+	int ret;
+	int val;
+
+	npages = ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift;
+	ASSERT_NE(npages, 0);
+	size = npages << self->page_shift;
+
+	buffer = hmm_buffer_alloc(size, size,
+				  PROT_READ | PROT_WRITE,
+				  MAP_PRIVATE | MAP_ANONYMOUS, -1);
+	ASSERT_NE(buffer, NULL);
+
+	/*
+	 * Initialize buffer in system memory but leave the first two pages
+	 * zero: page 0 stays pte_none so that hmm_range_fault() has to fault
+	 * it in, page 1 is mapped to the zero page.
+	 */
+	i = 2 * self->page_size / sizeof(*ptr);
+	for (ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
+		ptr[i] = i;
+
+	val = *(int *)(buffer->ptr + self->page_size);
+	ASSERT_EQ(val, 0);
+
+	/* Fault and migrate memory to device. */
+	ret = hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages);
+	ASSERT_EQ(ret, 0);
+	ASSERT_EQ(buffer->cpages, npages);
+
+	/* Check what the device read. */
+	for (i = 0, ptr = buffer->mirror; i < 2 * self->page_size / sizeof(*ptr); ++i)
+		ASSERT_EQ(ptr[i], 0);
+	for (; i < size / sizeof(*ptr); ++i)
+		ASSERT_EQ(ptr[i], i);
+
+	hmm_buffer_free(buffer);
+}
+
 /*
  * Migrate private file memory to device private memory.
  */
-- 
2.55.0

[PATCH v17 11/12] Documentation/mm/hmm: document migration through hmm_range_fault()

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Describe the two new MIGRATE_VMA_FAULT / MIGRATE_VMA_WRITE flags of
migrate_vma_setup(), and add a section on driving the migration
collection phase directly from hmm_range_fault() via HMM_PFN_REQ_MIGRATE
and migrate_hmm_range_setup().

Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 Documentation/mm/hmm.rst | 45 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index fc1b8dc19825..914ef6bfe848 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -348,6 +348,13 @@ between device driver specific code and shared common code:
    Currently only anonymous private VMA ranges can be migrated to or from
    system memory and device private memory.
 
+   By default only pages already present are collected. Two additional flags
+   ask migrate_vma_setup() to fault in missing pages first:
+
+   * ``MIGRATE_VMA_FAULT`` faults in missing pages with read access.
+   * ``MIGRATE_VMA_WRITE`` faults in missing pages with write access
+     (implies faulting).
+
    One of the first steps migrate_vma_setup() does is to invalidate other
    device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and
    ``mmu_notifier_invalidate_range_end()`` calls around the page table
@@ -427,6 +434,44 @@ between device driver specific code and shared common code:
 
    The lock can now be released.
 
+Migration collection through hmm_range_fault()
+==============================================
+
+The collection phase of migration (steps 1 and 2 above) can also be driven by
+hmm_range_fault() directly, sharing its page table walk. This lets a driver
+fault in and collect a range for migration in one walk, which is useful for
+migrate on fault.
+
+To do so, the driver sets ``HMM_PFN_REQ_MIGRATE`` in ``range->default_flags``
+and points ``range->migrate`` at a ``struct migrate_vma`` it has filled in
+(``flags``, ``src``, ``dst``, ``pgmap_owner``). Usually ``HMM_PFN_REQ_FAULT``
+(and ``HMM_PFN_REQ_WRITE``) is set as well, so missing pages are faulted in
+before being collected. The mmap_read_lock() has to be held for the whole
+migration, since the vma must stay stable.
+
+Migration requests come only from ``range->default_flags``:
+``range->pfn_flags_mask`` must be 0 and ``range->dev_private_owner`` must
+equal ``migrate->pgmap_owner``. ``range->hmm_pfns`` is cleared at the start of
+hmm_range_fault(), so per-pfn input, including ``HMM_PFN_INOUT_FLAGS`` such as
+``HMM_PFN_DMA_MAPPED``, is not preserved across a migrating call.
+
+hmm_range_fault() collects the entries the same way migrate_vma_setup() does,
+taking a folio reference, locking it and installing a migration PTE. Collected
+entries are marked with ``HMM_PFN_VALID | HMM_PFN_MIGRATE`` in
+``range->hmm_pfns``. If the page tables change while locks are dropped the
+partially collected entries are rolled back automatically.
+
+After hmm_range_fault() returns, the driver calls::
+
+    void migrate_hmm_range_setup(struct hmm_range *range);
+
+to translate ``range->hmm_pfns`` into ``migrate->src[]`` (``migrate->dst[]`` is
+zeroed) and initialize ``migrate->cpages`` and ``migrate->npages``. From here on
+the ``range->migrate`` struct is ready for the rest of the flow (steps 3
+onwards), i.e. migrate_vma_pages() and migrate_vma_finalize(). This should be
+called even on error, since hmm_range_fault() may have collected part of the
+range before failing.
+
 Exclusive access memory
 =======================
 
-- 
2.55.0

[PATCH v17 12/12] drm/nouveau/svm: add support for migrate on fault

[mpenttil]

From: Mika Penttilä <mpenttil@redhat.com>

Add a direct user for the new HMM_PFN_REQ_MIGRATE flag and
implement migrate on fault for nouveau driver. It is gated
by the svm_migrate_on_fault module parameter, default off.

Added nouveau_dmem_migrate_fault() that faults a range in and
does migrate collecting in a single hmm_range_fault() walk.

As an optimization, a migration policy - deciding which faulting
ranges are worth migrating to VRAM and which are better left
mirrored in system memory - would be useful. This patch
establishes the migrate-on-fault mechanism as a basis for later
refinement.

Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
 drivers/gpu/drm/nouveau/nouveau_dmem.c | 402 +++++++++++++++++++++++--
 drivers/gpu/drm/nouveau/nouveau_dmem.h |  12 +
 drivers/gpu/drm/nouveau/nouveau_svm.c  | 155 +++++++---
 drivers/gpu/drm/nouveau/nouveau_svm.h  |   5 +
 4 files changed, 503 insertions(+), 71 deletions(-)

diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c
index e74d7bb975a8..0a5d04068321 100644
--- a/drivers/gpu/drm/nouveau/nouveau_dmem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c
@@ -40,6 +40,9 @@
 #include <linux/hmm.h>
 #include <linux/memremap.h>
 #include <linux/migrate.h>
+#include <linux/jiffies.h>
+#include <linux/delay.h>
+#include <linux/mmu_notifier.h>
 
 /*
  * FIXME: this is ugly right now we are using TTM to allocate vram and we pin
@@ -76,6 +79,14 @@ struct nouveau_dmem_migrate {
 	nouveau_migrate_copy_t copy_func;
 	nouveau_clear_page_t clear_func;
 	struct nouveau_channel *chan;
+	/*
+	 * Serializes the dmem users of chan (CPU faults in migrate_to_ram(),
+	 * migrate on fault, bulk migration, eviction): the copy and clear
+	 * commands they write to its push buffer and the fences they emit.
+	 * Held only around those writes, never across page allocation.
+	 * TTM serializes its own use of the channel with cli->mutex.
+	 */
+	struct mutex lock;
 };
 
 struct nouveau_dmem {
@@ -93,6 +104,41 @@ struct nouveau_dmem_dma_info {
 	size_t size;
 };
 
+static int nouveau_dmem_copy(struct nouveau_drm *drm, u64 npages,
+			     enum nouveau_aper dst_aper, u64 dst_addr,
+			     enum nouveau_aper src_aper, u64 src_addr)
+{
+	int ret;
+
+	mutex_lock(&drm->dmem->migrate.lock);
+	ret = drm->dmem->migrate.copy_func(drm, npages, dst_aper, dst_addr,
+					   src_aper, src_addr);
+	mutex_unlock(&drm->dmem->migrate.lock);
+	return ret;
+}
+
+static int nouveau_dmem_clear(struct nouveau_drm *drm, u32 length,
+			      enum nouveau_aper dst_aper, u64 dst_addr)
+{
+	int ret;
+
+	mutex_lock(&drm->dmem->migrate.lock);
+	ret = drm->dmem->migrate.clear_func(drm, length, dst_aper, dst_addr);
+	mutex_unlock(&drm->dmem->migrate.lock);
+	return ret;
+}
+
+static int nouveau_dmem_fence_new(struct nouveau_drm *drm,
+				  struct nouveau_fence **fence)
+{
+	int ret;
+
+	mutex_lock(&drm->dmem->migrate.lock);
+	ret = nouveau_fence_new(fence, drm->dmem->migrate.chan);
+	mutex_unlock(&drm->dmem->migrate.lock);
+	return ret;
+}
+
 static struct nouveau_dmem_chunk *nouveau_page_to_chunk(struct page *page)
 {
 	return container_of(page_pgmap(page), struct nouveau_dmem_chunk,
@@ -139,10 +185,12 @@ static void nouveau_dmem_folio_free(struct folio *folio)
 	spin_unlock(&dmem->lock);
 }
 
-static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
+static int nouveau_dmem_fence_done(struct nouveau_fence **fence)
 {
+	int ret = 0;
+
 	if (fence) {
-		nouveau_fence_wait(*fence, true, false);
+		ret = nouveau_fence_wait(*fence, true, false);
 		nouveau_fence_unref(fence);
 	} else {
 		/*
@@ -150,6 +198,54 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
 		 * the hmem object.
 		 */
 	}
+
+	/* 0 on completion, -EBUSY on timeout, or other errno; propagated. */
+	return ret;
+}
+
+/*
+ * Poll the copy fence instead of the lazy (interrupt-driven) wait since
+ * completion IRQ is randomly missed, hitting the timeout.
+ *
+ * Returns 0 on completion, -EBUSY on timeout. Consumes the fence reference.
+ */
+static int nouveau_dmem_fence_poll(struct nouveau_fence **fence)
+{
+	unsigned long deadline = jiffies + 30 * HZ;
+	int ret = 0;
+
+	while (!nouveau_fence_done(*fence)) {
+		if (time_after(jiffies, deadline)) {
+			ret = -EBUSY;
+			break;
+		}
+		schedule_timeout_uninterruptible(msecs_to_jiffies(1));
+	}
+	nouveau_fence_unref(fence);
+
+	return ret;
+}
+
+/*
+ * The copy could not be confirmed (fence timeout or no fence): don't migrate,
+ * so finalize restores the original ptes and the source stays valid. The copy
+ * engine may still be using the pages, so keep the destination folios (an
+ * extra reference that finalize won't drop) and let the caller skip the DMA
+ * unmap: leaking is safer than freeing memory a wedged engine might write.
+ */
+static void nouveau_dmem_migrate_abort(struct nouveau_drm *drm,
+				       struct migrate_vma *args)
+{
+	unsigned long i;
+
+	for (i = 0; i < args->npages; i++) {
+		struct page *dpage = migrate_pfn_to_page(args->dst[i]);
+
+		args->src[i] &= ~MIGRATE_PFN_MIGRATE;
+		if (dpage)
+			folio_get(page_folio(dpage));
+	}
+	NV_ERROR(drm, "svm: copy not confirmed, migration aborted, pages leaked\n");
 }
 
 static int nouveau_dmem_copy_folio(struct nouveau_drm *drm,
@@ -168,10 +264,10 @@ static int nouveau_dmem_copy_folio(struct nouveau_drm *drm,
 	if (dma_mapping_error(dev, dma_info->dma_addr))
 		return -EIO;
 
-	if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(sfolio),
-					 NOUVEAU_APER_HOST, dma_info->dma_addr,
-					 NOUVEAU_APER_VRAM,
-					 nouveau_dmem_page_addr(spage))) {
+	if (nouveau_dmem_copy(drm, folio_nr_pages(sfolio),
+			      NOUVEAU_APER_HOST, dma_info->dma_addr,
+			      NOUVEAU_APER_VRAM,
+			      nouveau_dmem_page_addr(spage))) {
 		dma_unmap_page(dev, dma_info->dma_addr, page_size(dpage),
 					DMA_BIDIRECTIONAL);
 		return -EIO;
@@ -183,12 +279,11 @@ static int nouveau_dmem_copy_folio(struct nouveau_drm *drm,
 static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
 {
 	struct nouveau_drm *drm = page_to_drm(vmf->page);
-	struct nouveau_dmem *dmem = drm->dmem;
-	struct nouveau_fence *fence;
+	struct nouveau_fence *fence = NULL;
 	struct nouveau_svmm *svmm;
 	struct page *dpage;
 	vm_fault_t ret = 0;
-	int err;
+	int err, fret = 0;
 	struct migrate_vma args = {
 		.vma		= vmf->vma,
 		.pgmap_owner	= drm->dev,
@@ -233,10 +328,12 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
 	 * than just one page on CPU fault. When such fault happens it is very
 	 * likely that more surrounding page will CPU fault too.
 	 */
-	if (migrate_vma_setup(&args) < 0)
-		return VM_FAULT_SIGBUS;
+	if (migrate_vma_setup(&args) < 0) {
+		ret = VM_FAULT_SIGBUS;
+		goto err;
+	}
 	if (!args.cpages)
-		return 0;
+		goto done;
 
 	if (order)
 		dpage = folio_page(vma_alloc_folio(GFP_HIGHUSER | __GFP_ZERO,
@@ -264,11 +361,21 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
 		goto done;
 	}
 
-	nouveau_fence_new(&fence, dmem->migrate.chan);
+	/*
+	 * Commit the migration only once the copy is confirmed. Without a
+	 * fence the copy can't be confirmed: treat it as failed.
+	 */
+	fret = nouveau_dmem_fence_new(drm, &fence);
+	if (!fret)
+		fret = nouveau_dmem_fence_poll(&fence);
+	if (fret) {
+		nouveau_dmem_migrate_abort(drm, &args);
+		ret = VM_FAULT_SIGBUS;
+	} else {
+		dma_unmap_page(drm->dev->dev, dma_info.dma_addr, dma_info.size,
+			       DMA_BIDIRECTIONAL);
+	}
 	migrate_vma_pages(&args);
-	nouveau_dmem_fence_done(&fence);
-	dma_unmap_page(drm->dev->dev, dma_info.dma_addr, dma_info.size,
-				DMA_BIDIRECTIONAL);
 done:
 	migrate_vma_finalize(&args);
 err:
@@ -528,7 +635,7 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
 		}
 	}
 
-	nouveau_fence_new(&fence, chunk->drm->dmem->migrate.chan);
+	nouveau_dmem_fence_new(chunk->drm, &fence);
 	migrate_device_pages(src_pfns, dst_pfns, npages);
 	nouveau_dmem_fence_done(&fence);
 	migrate_device_finalize(src_pfns, dst_pfns, npages);
@@ -729,6 +836,7 @@ nouveau_dmem_init(struct nouveau_drm *drm)
 	INIT_LIST_HEAD(&drm->dmem->chunks);
 	mutex_init(&drm->dmem->mutex);
 	spin_lock_init(&drm->dmem->lock);
+	mutex_init(&drm->dmem->migrate.lock);
 
 	/* Initialize migration dma helpers before registering memory */
 	ret = nouveau_dmem_migrate_init(drm);
@@ -764,13 +872,13 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
 		dma_info->size = page_size(spage);
 		if (dma_mapping_error(dev, dma_info->dma_addr))
 			goto out_free_page;
-		if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(page_folio(spage)),
+		if (nouveau_dmem_copy(drm, folio_nr_pages(page_folio(spage)),
 			NOUVEAU_APER_VRAM, paddr, NOUVEAU_APER_HOST,
 			dma_info->dma_addr))
 			goto out_dma_unmap;
 	} else {
 		dma_info->dma_addr = DMA_MAPPING_ERROR;
-		if (drm->dmem->migrate.clear_func(drm, page_size(dpage),
+		if (nouveau_dmem_clear(drm, page_size(dpage),
 			NOUVEAU_APER_VRAM, paddr))
 			goto out_free_page;
 	}
@@ -794,11 +902,12 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
 	return 0;
 }
 
-static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
+static unsigned long
+nouveau_dmem_migrate_alloc_and_copy(struct nouveau_drm *drm,
 		struct nouveau_svmm *svmm, struct migrate_vma *args,
-		struct nouveau_dmem_dma_info *dma_info, u64 *pfns)
+		struct nouveau_dmem_dma_info *dma_info, u64 *pfns,
+		unsigned long *nr_dma_out, unsigned long *order_out)
 {
-	struct nouveau_fence *fence;
 	unsigned long addr = args->start, nr_dma = 0, i;
 	unsigned long order = 0;
 
@@ -820,7 +929,22 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
 		addr += (1 << order) * PAGE_SIZE;
 	}
 
-	nouveau_fence_new(&fence, drm->dmem->migrate.chan);
+	*nr_dma_out = nr_dma;
+	*order_out = order;
+	return i;
+}
+
+static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
+		struct nouveau_svmm *svmm, struct migrate_vma *args,
+		struct nouveau_dmem_dma_info *dma_info, u64 *pfns)
+{
+	struct nouveau_fence *fence;
+	unsigned long nr_dma, order, i;
+
+	i = nouveau_dmem_migrate_alloc_and_copy(drm, svmm, args, dma_info, pfns,
+						&nr_dma, &order);
+
+	nouveau_dmem_fence_new(drm, &fence);
 	migrate_vma_pages(args);
 	nouveau_dmem_fence_done(&fence);
 	nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i, order);
@@ -832,6 +956,238 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
 	migrate_vma_finalize(args);
 }
 
+/*
+ * Encode an hmm_range_fault() pfn for @start into the nouveau PFNMAP format,
+ * shared by nouveau_hmm_convert_pfn() and the in-place map in
+ * nouveau_dmem_migrate_fault().
+ */
+u64 nouveau_dmem_hmm_pfn_to_map(unsigned long hmm_pfn, unsigned long start,
+				unsigned long *map_addr, unsigned int *page_shift)
+{
+	unsigned long order;
+	struct page *page;
+	u64 pfn;
+
+	*map_addr = start;
+	*page_shift = PAGE_SHIFT;
+
+	if (!(hmm_pfn & HMM_PFN_VALID))
+		return NVIF_VMM_PFNMAP_V0_NONE;
+
+	page = hmm_pfn_to_page(hmm_pfn);
+
+	order = hmm_pfn_to_map_order(hmm_pfn);
+	if (order) {
+		*page_shift = order + PAGE_SHIFT;
+		*map_addr = start & ~((1UL << *page_shift) - 1);
+		page -= (start - *map_addr) >> PAGE_SHIFT;
+	}
+
+	if (is_device_private_page(page))
+		pfn = nouveau_dmem_page_addr(page) |
+			NVIF_VMM_PFNMAP_V0_V | NVIF_VMM_PFNMAP_V0_VRAM;
+	else
+		pfn = page_to_phys(page) |
+			NVIF_VMM_PFNMAP_V0_V | NVIF_VMM_PFNMAP_V0_HOST;
+	if (hmm_pfn & HMM_PFN_WRITE)
+		pfn |= NVIF_VMM_PFNMAP_V0_W;
+
+	return pfn;
+}
+
+/*
+ * Fault a range in and migrate it to VRAM.
+ * Returns 0 on success, or an error (so the caller
+ * falls back to nouveau_range_fault()) if nothing could be mapped.
+ */
+int
+nouveau_dmem_migrate_fault(struct nouveau_drm *drm, struct nouveau_svmm *svmm,
+			   struct mmu_interval_notifier *notifier,
+			   unsigned long start, unsigned long end,
+			   unsigned long hmm_flags,
+			   u64 *out_pfn0, unsigned long *out_addr,
+			   unsigned int *out_shift)
+{
+	unsigned long timeout =
+		jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+	unsigned long npages = (end - start) >> PAGE_SHIFT;
+	struct mm_struct *mm = notifier->mm;
+	struct nouveau_dmem_dma_info *dma_info;
+	struct nouveau_fence *fence = NULL;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= drm->dev,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct hmm_range range = {
+		.notifier		= notifier,
+		.start			= start,
+		.end			= end,
+		.dev_private_owner	= drm->dev,
+		.migrate		= &migrate,
+		.default_flags		= hmm_flags | HMM_PFN_REQ_MIGRATE,
+	};
+	unsigned long nr_dma, order, n, i;
+	unsigned long hmm_pfn0 = 0;
+	bool migrated;
+	u64 *pfns;
+	int ret, fret = 0;
+
+	if (!drm->dmem)
+		return -ENODEV;
+
+	migrate.src = kcalloc(npages, sizeof(*migrate.src), GFP_KERNEL);
+	if (!migrate.src)
+		return -ENOMEM;
+	migrate.dst = kcalloc(npages, sizeof(*migrate.dst), GFP_KERNEL);
+	if (!migrate.dst) {
+		ret = -ENOMEM;
+		goto out_free_src;
+	}
+	dma_info = kmalloc_objs(*dma_info, npages);
+	if (!dma_info) {
+		ret = -ENOMEM;
+		goto out_free_dst;
+	}
+	pfns = nouveau_pfns_alloc(npages);
+	if (!pfns) {
+		ret = -ENOMEM;
+		goto out_free_dma;
+	}
+
+	/* hmm_range_fault() shares the source pfn array with migrate_vma. */
+	range.hmm_pfns = migrate.src;
+
+	do {
+		mmap_read_lock(mm);
+		range.notifier_seq = mmu_interval_read_begin(notifier);
+
+		ret = hmm_range_fault(&range);
+		/*
+		 * Snapshot the pfn before migrate_hmm_range_setup() overwrites the
+		 * shared array; used to map in place if nothing migrates.
+		 */
+		if (!ret)
+			hmm_pfn0 = range.hmm_pfns[0];
+		migrate_hmm_range_setup(&range);
+		if (ret) {
+			migrate_vma_pages(&migrate);
+			migrate_vma_finalize(&migrate);
+			goto unlock;
+		}
+
+		n = nouveau_dmem_migrate_alloc_and_copy(drm, svmm, &migrate,
+							dma_info, pfns,
+							&nr_dma, &order);
+
+		/* Only a migrated page has an in-flight CE copy to unblock below. */
+		for (i = 0, migrated = false; i < npages; i++) {
+			if (migrate.dst[i]) {
+				migrated = true;
+				break;
+			}
+		}
+
+		/*
+		 * Track a copy fence only when a copy was enqueued; waiting on an
+		 * empty fence would block until the fault's CTXSW_TIMEOUT.
+		 */
+		if (migrated)
+			fret = nouveau_dmem_fence_new(drm, &fence);
+
+		/*
+		 * The fault-in path must replay the faulting channel before waiting on
+		 * the copy fence: with the fault still unreplayed the copy does not
+		 * make progress. GPU has no mapping so it might refault, until it
+		 * gets the migrated or mirrored mapping.
+		 */
+		if (drm->svm && migrated)
+			nouveau_svm_fault_replay(drm->svm);
+
+		/* Without a fence (fret set) the copy can't be confirmed. */
+		if (migrated && !fret)
+			fret = nouveau_dmem_fence_poll(&fence);
+
+		/* Commit the migration only once the copy is confirmed. */
+		if (fret)
+			nouveau_dmem_migrate_abort(drm, &migrate);
+		migrate_vma_pages(&migrate);
+
+		/*
+		 * Don't map destination pages that migrate_vma_pages() didn't
+		 * migrate: finalize frees them and restores the source.
+		 */
+		for (i = 0; i < npages; i++)
+			if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+				pfns[i] = NVIF_VMM_PFNMAP_V0_NONE;
+
+		mutex_lock(&svmm->mutex);
+		if (fret) {
+			/* Copy did not complete; don't map a half-copied page. */
+			ret = -EIO;
+		} else if (mmu_interval_read_retry(notifier, range.notifier_seq)) {
+			ret = -EBUSY;
+		} else {
+			unsigned long map_addr = migrate.start;
+			unsigned int page_shift = PAGE_SHIFT + order;
+
+			/*
+			 * If nothing migrated might still be able to mirror.
+			 * In that case use the snapshot hmm_pfn.
+			 * npages == 1 always today.
+			 */
+			if (npages == 1 && !(pfns[0] & NVIF_VMM_PFNMAP_V0_V))
+				pfns[0] = nouveau_dmem_hmm_pfn_to_map(hmm_pfn0,
+						start, &map_addr, &page_shift);
+
+			/*
+			 * Nothing to map: leave the GPU mapping alone, the
+			 * caller falls back to nouveau_range_fault().
+			 */
+			if (pfns[0] & NVIF_VMM_PFNMAP_V0_V) {
+				/* On failure the caller falls back, then cancels */
+				ret = __nouveau_pfns_map_locked(svmm, map_addr,
+								pfns, n,
+								page_shift);
+				if (!ret) {
+					*out_pfn0 = pfns[0];
+					*out_addr = map_addr;
+					*out_shift = page_shift;
+				}
+			}
+		}
+		mutex_unlock(&svmm->mutex);
+
+		if (fret)
+			NV_ERROR(drm,
+				 "svm: migrate copy did not complete (%d) for %lx-%lx\n",
+				 fret, start, end);
+
+		/* After an aborted copy the engine may still use the mappings */
+		while (!fret && nr_dma--) {
+			dma_unmap_page(drm->dev->dev, dma_info[nr_dma].dma_addr,
+				       dma_info[nr_dma].size, DMA_BIDIRECTIONAL);
+		}
+		migrate_vma_finalize(&migrate);
+unlock:
+		mmap_read_unlock(mm);
+	} while (ret == -EBUSY && !time_after(jiffies, timeout));
+
+	/* Nothing migrated or mappable in place, fallback to nouveau_range_fault(). */
+	if (!ret && !(pfns[0] & NVIF_VMM_PFNMAP_V0_V))
+		ret = -ENOENT;
+
+	nouveau_pfns_free(pfns);
+out_free_dma:
+	kfree(dma_info);
+out_free_dst:
+	kfree(migrate.dst);
+out_free_src:
+	kfree(migrate.src);
+	return ret;
+}
+
 int
 nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
 			 struct nouveau_svmm *svmm,
diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.h b/drivers/gpu/drm/nouveau/nouveau_dmem.h
index 64da5d3635c8..91e8910cadbf 100644
--- a/drivers/gpu/drm/nouveau/nouveau_dmem.h
+++ b/drivers/gpu/drm/nouveau/nouveau_dmem.h
@@ -27,6 +27,7 @@ struct drm_file;
 struct nouveau_drm;
 struct nouveau_svmm;
 struct hmm_range;
+struct mmu_interval_notifier;
 
 #if IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM)
 void nouveau_dmem_init(struct nouveau_drm *);
@@ -39,7 +40,18 @@ int nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
 			     struct vm_area_struct *vma,
 			     unsigned long start,
 			     unsigned long end);
+int nouveau_dmem_migrate_fault(struct nouveau_drm *drm,
+			       struct nouveau_svmm *svmm,
+			       struct mmu_interval_notifier *notifier,
+			       unsigned long start,
+			       unsigned long end,
+			       unsigned long hmm_flags,
+			       u64 *out_pfn0,
+			       unsigned long *out_addr,
+			       unsigned int *out_shift);
 unsigned long nouveau_dmem_page_addr(struct page *page);
+u64 nouveau_dmem_hmm_pfn_to_map(unsigned long hmm_pfn, unsigned long start,
+				unsigned long *map_addr, unsigned int *page_shift);
 
 #else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */
 static inline void nouveau_dmem_init(struct nouveau_drm *drm) {}
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
index c519c28164fa..6e109274bd77 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
@@ -37,6 +37,7 @@
 #include <linux/hmm.h>
 #include <linux/memremap.h>
 #include <linux/rmap.h>
+#include <linux/moduleparam.h>
 
 struct nouveau_svm {
 	struct nouveau_drm *drm;
@@ -78,6 +79,13 @@ struct nouveau_svm {
 #define SVM_DBG(s,f,a...) NV_DEBUG((s)->drm, "svm: "f"\n", ##a)
 #define SVM_ERR(s,f,a...) NV_WARN((s)->drm, "svm: "f"\n", ##a)
 
+/* When set, migrate the faulting range to VRAM on GPU fault (default off). */
+static bool nouveau_svm_migrate_on_fault;
+module_param_named(svm_migrate_on_fault, nouveau_svm_migrate_on_fault, bool,
+		   0644);
+MODULE_PARM_DESC(svm_migrate_on_fault,
+		 "migrate system memory to VRAM on GPU fault (default off)");
+
 struct nouveau_pfnmap_args {
 	struct nvif_ioctl_v0_hdr i;
 	struct nvif_ioctl_mthd_v0_hdr m;
@@ -182,9 +190,10 @@ nouveau_svmm_bind(struct drm_device *dev, void *data,
 
 		addr = max(addr, vma->vm_start);
 		next = min(vma->vm_end, end);
-		/* This is a best effort so we ignore errors */
-		nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma, addr,
-					 next);
+		/* Best effort; skipped when migrate-on-fault does the migration. */
+		if (!nouveau_svm_migrate_on_fault)
+			nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma,
+						 addr, next);
 		addr = next;
 	}
 
@@ -390,7 +399,7 @@ nouveau_svmm_init(struct drm_device *dev, void *data,
 }
 
 /* Issue fault replay for GPU to retry accesses that faulted previously. */
-static void
+void
 nouveau_svm_fault_replay(struct nouveau_svm *svm)
 {
 	SVM_DBG(svm, "replay");
@@ -514,6 +523,8 @@ nouveau_svm_fault_cache(struct nouveau_svm *svm,
 struct svm_notifier {
 	struct mmu_interval_notifier notifier;
 	struct nouveau_svmm *svmm;
+	/* Task running this fault's migrate-on-fault walk, or NULL */
+	struct task_struct *migrating;
 };
 
 static bool nouveau_svm_range_invalidate(struct mmu_interval_notifier *mni,
@@ -527,6 +538,17 @@ static bool nouveau_svm_range_invalidate(struct mmu_interval_notifier *mni,
 	    range->owner == sn->svmm->vmm->cli->drm->dev)
 		return true;
 
+	/*
+	 * Ignore the invalidation of this fault's own migrate-on-fault walk,
+	 * which runs in the walking task; otherwise its MMU_NOTIFY_MIGRATE
+	 * bumps the seq and read_retry() spins to timeout. Other migrations
+	 * on the device (CPU faults, bind) must still invalidate.
+	 */
+	if (range->event == MMU_NOTIFY_MIGRATE &&
+	    range->owner == sn->svmm->vmm->cli->drm->dev &&
+	    READ_ONCE(sn->migrating) == current)
+		return true;
+
 	/*
 	 * serializes the update to mni->invalidate_seq done by caller and
 	 * prevents invalidation of the PTE from progressing while HW is being
@@ -551,48 +573,25 @@ static void nouveau_hmm_convert_pfn(struct nouveau_drm *drm,
 				    struct hmm_range *range,
 				    struct nouveau_pfnmap_args *args)
 {
-	struct page *page;
+	unsigned long map_addr;
+	unsigned int page_shift;
 
 	/*
 	 * The address prepared here is passed through nvif_object_ioctl()
 	 * to an eventual DMA map in something like gp100_vmm_pgt_pfn()
 	 *
 	 * This is all just encoding the internal hmm representation into a
-	 * different nouveau internal representation.
+	 * different nouveau internal representation; see the shared encoder
+	 * nouveau_dmem_hmm_pfn_to_map() (also widens to the CPU folio order).
 	 */
-	if (!(range->hmm_pfns[0] & HMM_PFN_VALID)) {
-		args->p.phys[0] = 0;
-		return;
+	args->p.phys[0] = nouveau_dmem_hmm_pfn_to_map(range->hmm_pfns[0],
+						      args->p.addr, &map_addr,
+						      &page_shift);
+	if (page_shift != PAGE_SHIFT) {
+		args->p.addr = map_addr;
+		args->p.page = page_shift;
+		args->p.size = 1UL << page_shift;
 	}
-
-	page = hmm_pfn_to_page(range->hmm_pfns[0]);
-	/*
-	 * Only map compound pages to the GPU if the CPU is also mapping the
-	 * page as a compound page. Otherwise, the PTE protections might not be
-	 * consistent (e.g., CPU only maps part of a compound page).
-	 * Note that the underlying page might still be larger than the
-	 * CPU mapping (e.g., a PUD sized compound page partially mapped with
-	 * a PMD sized page table entry).
-	 */
-	if (hmm_pfn_to_map_order(range->hmm_pfns[0])) {
-		unsigned long addr = args->p.addr;
-
-		args->p.page = hmm_pfn_to_map_order(range->hmm_pfns[0]) +
-				PAGE_SHIFT;
-		args->p.size = 1UL << args->p.page;
-		args->p.addr &= ~(args->p.size - 1);
-		page -= (addr - args->p.addr) >> PAGE_SHIFT;
-	}
-	if (is_device_private_page(page))
-		args->p.phys[0] = nouveau_dmem_page_addr(page) |
-				NVIF_VMM_PFNMAP_V0_V |
-				NVIF_VMM_PFNMAP_V0_VRAM;
-	else
-		args->p.phys[0] = page_to_phys(page) |
-				NVIF_VMM_PFNMAP_V0_V |
-				NVIF_VMM_PFNMAP_V0_HOST;
-	if (range->hmm_pfns[0] & HMM_PFN_WRITE)
-		args->p.phys[0] |= NVIF_VMM_PFNMAP_V0_W;
 }
 
 static int nouveau_atomic_range_fault(struct nouveau_svmm *svmm,
@@ -728,6 +727,46 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
 	return ret;
 }
 
+static int nouveau_range_fault_and_migrate(struct nouveau_svmm *svmm,
+					   struct nouveau_drm *drm,
+					   struct nouveau_pfnmap_args *args,
+					   unsigned long hmm_flags,
+					   struct svm_notifier *notifier)
+{
+	struct mm_struct *mm = svmm->notifier.mm;
+	unsigned long start = args->p.addr;
+	unsigned long end = start + args->p.size;
+	unsigned long map_addr;
+	unsigned int page_shift;
+	u64 pfn0;
+	int ret;
+
+	ret = mmu_interval_notifier_insert(&notifier->notifier, mm,
+					   start, args->p.size,
+					   &nouveau_svm_mni_ops);
+	if (ret)
+		return ret;
+
+	WRITE_ONCE(notifier->migrating, current);
+	ret = nouveau_dmem_migrate_fault(drm, svmm, &notifier->notifier,
+					 start, end, hmm_flags,
+					 &pfn0, &map_addr, &page_shift);
+	WRITE_ONCE(notifier->migrating, NULL);
+
+	if (!ret) {
+		args->p.phys[0] = pfn0;
+		if (page_shift != PAGE_SHIFT) {
+			args->p.addr = map_addr;
+			args->p.page = page_shift;
+			args->p.size = 1UL << page_shift;
+		}
+	}
+
+	mmu_interval_notifier_remove(&notifier->notifier);
+
+	return ret;
+}
+
 static void
 nouveau_svm_fault(struct work_struct *work)
 {
@@ -843,14 +882,26 @@ nouveau_svm_fault(struct work_struct *work)
 		}
 
 		notifier.svmm = svmm;
-		if (atomic)
+		notifier.migrating = NULL;
+		if (atomic) {
 			ret = nouveau_atomic_range_fault(svmm, svm->drm, args,
 							 __struct_size(args),
 							 &notifier);
-		else
+		} else if (nouveau_svm_migrate_on_fault &&
+			   buffer->fault[fi]->access != FAULT_ACCESS_PREFETCH) {
+			/* Fault+migrate in one walk; fall back if nothing migrated. */
+			ret = nouveau_range_fault_and_migrate(svmm, svm->drm,
+							      args, hmm_flags,
+							      &notifier);
+			if (ret)
+				ret = nouveau_range_fault(svmm, svm->drm, args,
+							  __struct_size(args),
+							  hmm_flags, &notifier);
+		} else {
 			ret = nouveau_range_fault(svmm, svm->drm, args,
 						  __struct_size(args),
 						  hmm_flags, &notifier);
+		}
 		mmput(mm);
 
 		limit = args->p.addr + args->p.size;
@@ -934,22 +985,30 @@ nouveau_pfns_free(u64 *pfns)
 	kfree(args);
 }
 
-void
-nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
-		 unsigned long addr, u64 *pfns, unsigned long npages,
-		 unsigned int page_shift)
+int
+__nouveau_pfns_map_locked(struct nouveau_svmm *svmm, unsigned long addr,
+			  u64 *pfns, unsigned long npages,
+			  unsigned int page_shift)
 {
 	struct nouveau_pfnmap_args *args = nouveau_pfns_to_args(pfns);
 
+	lockdep_assert_held(&svmm->mutex);
+
 	args->p.addr = addr;
 	args->p.size = npages << page_shift;
 	args->p.page = page_shift;
 
-	mutex_lock(&svmm->mutex);
-
-	nvif_object_ioctl(&svmm->vmm->vmm.object, args,
-			  struct_size(args, p.phys, npages), NULL);
+	return nvif_object_ioctl(&svmm->vmm->vmm.object, args,
+				 struct_size(args, p.phys, npages), NULL);
+}
 
+void
+nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
+		 unsigned long addr, u64 *pfns, unsigned long npages,
+		 unsigned int page_shift)
+{
+	mutex_lock(&svmm->mutex);
+	__nouveau_pfns_map_locked(svmm, addr, pfns, npages, page_shift);
 	mutex_unlock(&svmm->mutex);
 }
 
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.h b/drivers/gpu/drm/nouveau/nouveau_svm.h
index 3fd78662f17e..c24b717819a5 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.h
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.h
@@ -5,6 +5,7 @@
 struct drm_device;
 struct drm_file;
 struct nouveau_drm;
+struct nouveau_svm;
 
 struct nouveau_svmm {
 	struct mmu_notifier notifier;
@@ -35,6 +36,10 @@ void nouveau_pfns_free(u64 *pfns);
 void nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
 		      unsigned long addr, u64 *pfns, unsigned long npages,
 		      unsigned int page_shift);
+int __nouveau_pfns_map_locked(struct nouveau_svmm *svmm, unsigned long addr,
+			      u64 *pfns, unsigned long npages,
+			      unsigned int page_shift);
+void nouveau_svm_fault_replay(struct nouveau_svm *svm);
 #else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */
 static inline void nouveau_svm_init(struct nouveau_drm *drm) {}
 static inline void nouveau_svm_fini(struct nouveau_drm *drm) {}
-- 
2.55.0

Re: [PATCH v17 03/12] mm/hmm: preparations for HMM to participate in migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm/hmm: memset() in hmm_range_fault_locked() wipes collected pfns on retry
- [Low] mm/hmm: build failure in hmm_range_fault_locked() when !CONFIG_MIGRATION
--

commit 9e255c525419b859ce3c5e39d7de2c6153874013
Author: Mika Penttilä <mpenttil@redhat.com>

mm/hmm: preparations for HMM to participate in migration

This commit prepares hmm_range_fault() to participate in page migration by
capturing the VMA during page table walks, managing MMU_NOTIFY_MIGRATE
invalidations, and clearing range->hmm_pfns at the start of migrating
faults.

> diff --git a/mm/hmm.c b/mm/hmm.c
> index 2f1e98c6b6440..9220e6648636d 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c

[ ... ]

> @@ -694,19 +774,45 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
>  {
>  	struct hmm_vma_walk hmm_vma_walk = {
>  		.range = range,
> -		.locked = locked,
> +		.mmlocked = locked,
>  		.last = range->start,
>  	};
> -	struct mm_struct *mm = range->notifier->mm;
> +	/*
> +	 *  Could be serving a device fault or come from migrate
> +	 *  entry point. For the former we have not resolved the vma
> +	 *  yet, and the latter we don't have a notifier (but have a vma).
> +	 *
> +	 */
> +	struct mm_struct *mm = hmm_range_fault_mm(range);
>  	int ret;
>  
> +	/* Migration is incompatible with mmap lock drop */
> +	if (locked && hmm_select_migrate(range))
> +		return -EINVAL;
> +
> +	/*
> +	 * Migration takes its requests only from default_flags, so the input
> +	 * pfns carry nothing. Clear them so that collected entries can be
> +	 * told apart reliably during the walk and on rollback.
> +	 */
> +	if (hmm_select_migrate(range)) {
> +		WARN_ON_ONCE(range->pfn_flags_mask);
> +		WARN_ON_ONCE(range->dev_private_owner !=
> +			     range->migrate->pgmap_owner);

[Severity: Low]
Will this break the build when CONFIG_HMM_MIRROR=y and CONFIG_MIGRATION=n?

Several Kconfig options (such as CONFIG_INFINIBAND_ON_DEMAND_PAGING and
CONFIG_DRM_AMDGPU_USERPTR) select CONFIG_HMM_MIRROR without depending on
CONFIG_MIGRATION. In include/linux/hmm.h, struct migrate_vma is only
forward-declared, and in include/linux/migrate.h, its definition is guarded
by #ifdef CONFIG_MIGRATION while a stub is provided for
hmm_select_migrate() when !CONFIG_MIGRATION.

Even though hmm_select_migrate(range) returns 0 when CONFIG_MIGRATION is
disabled, the compiler still type-checks range->migrate->pgmap_owner inside
hmm_range_fault_locked() in mm/hmm.c and fails due to dereferencing a
pointer to an incomplete type.

Should this check be moved into a helper in include/linux/migrate.h with a
matching !CONFIG_MIGRATION stub?

> +		memset(range->hmm_pfns, 0,
> +		       ((range->end - range->start) >> PAGE_SHIFT) *
> +		       sizeof(*range->hmm_pfns));
> +	}
> +
>  	mmap_assert_locked(mm);
>  
>  	do {
>  		/* If range is no longer valid force retry. */
> -		if (mmu_interval_check_retry(range->notifier,
> -					     range->notifier_seq))
> -			return -EBUSY;
> +		if (range->notifier && mmu_interval_check_retry(range->notifier,
> +								range->notifier_seq)) {
> +			ret =  -EBUSY;
> +			break;
> +		}

[Severity: High]
Can unconditionally calling memset() on range->hmm_pfns in
hmm_range_fault_locked() cause a self-deadlock on the folio lock or leak
locked folios when a migrating hmm_range_fault() call is retried after
-EBUSY?

When migration is enabled in this series with range->notifier set (such as
in dmirror_range_fault() in lib/test_hmm.c), walk_page_range() collects
present pages by locking each folio, elevating its reference count,
installing a migration PTE, and marking range->hmm_pfns[i] with
HMM_PFN_VALID | HMM_PFN_MIGRATE.

If a later page in the range needs a fault, walk_page_range() returns
HMM_FAULT_PENDING, hmm_vma_migrate_disarm() closes the MMU_NOTIFY_MIGRATE
invalidation, and hmm_do_fault() calls handle_mm_fault() while the
previously collected folios stay locked by current and mapped as migration
PTEs. If handle_mm_fault() or a concurrent operation triggers an MMU
notifier invalidation, mmu_interval_check_retry() on the next loop
iteration returns true and hmm_range_fault_locked() exits with -EBUSY (or
the caller retries after mmu_interval_read_retry() fails).

When the caller retries hmm_range_fault(range) with the same
range->hmm_pfns array, the unconditional memset(range->hmm_pfns, 0, ...) at
the top of hmm_range_fault_locked() wipes the HMM_PFN_VALID |
HMM_PFN_MIGRATE markers for the folios that are still locked and installed
as migration PTEs. On the retry walk, hmm_pfn_collected() will return false
for those entries, causing hmm_vma_handle_pte() or hmm_vma_walk_pmd() to
treat current's own migration entries as another task's migration entries
and wait in migration_entry_wait() on a folio already locked by current
(or leak the locked folios and migration PTEs if the caller aborts).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=3

Re: [PATCH v17 04/12] mm/hmm: do the plumbing for HMM to participate in migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm/hmm: hmm_vma_walk_pmd() skips remaining PTEs on folio split failure
--

commit 467255657c2d81a197c74dccb6095cef3e48051a
Author: Mika Penttilä <mpenttil@redhat.com>

mm/hmm: do the plumbing for HMM to participate in migration

This commit prepares hmm_range_fault() and the HMM pagewalk callbacks for
migration collection by adding PMD and PTE locking during the walk and wiring
in the migration preparation hooks.

> diff --git a/mm/hmm.c b/mm/hmm.c
> index 9220e6648636d..854d2c090b20c 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c

[ ... ]

> @@ -532,24 +739,65 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
>  	 * recover.
>  	 */
>  	if (pmd_bad(pmd)) {
> -		if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0))
> +		if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0))
>  			return -EFAULT;
> -		return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
> +		return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
>  	}
>  
> -	ptep = pte_offset_map(pmdp, addr);
> -	if (!ptep)
> +	if (minfo) {
> +		ptep = pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl);
> +		if (ptep)
> +			hmm_vma_walk->ptelocked = true;
> +	} else {
> +		ptep = pte_offset_map(pmdp, addr);
> +	}
> +	if (!ptep) {
> +		addr = start;
>  		goto again;
> +	}
> +
>  	for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) {
> -		int r;
>  
>  		r = hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns);
>  		if (r) {
> -			/* hmm_vma_handle_pte() did pte_unmap() */
> +			/* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */
> +			if (unmapped)
> +				flush_tlb_range(walk->vma, start, addr);
>  			return r;
>  		}
> +
> +		r = hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &unmapped);
> +		if (r == -EAGAIN) {
> +			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
> +			if (unmapped) {
> +				flush_tlb_range(walk->vma, start, addr);
> +				unmapped = false;
> +			}
> +			goto again;
> +		}
> +		if (r) {
> +			/*
> +			 * Splitting the folio failed; migrate_vma_split_folio()
> +			 * already dropped the PTE lock. Don't migrate the rest
> +			 * of the pmd range; addr keeps its valid pfn.
> +			 */
> +			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
> +			if (unmapped)
> +				flush_tlb_range(walk->vma, start, addr);
> +			return 0;
> +		}

[Severity: Medium]
As reported by Sashiko on v16 [1], when hmm_vma_handle_migrate_prepare()
returns a non-zero error other than -EAGAIN in hmm_vma_walk_pmd() (such as
when migrate_vma_split_folio() fails to split a PTE-mapped large folio in
the subsequent patch "mm/hmm: migrate collection in HMM pagewalk - pte
level"), does returning 0 here leave the remaining entries in
[addr + PAGE_SIZE, end) completely unprocessed?

At the start of a migrating walk, hmm_range_fault_locked() clears
range->hmm_pfns[]:

mm/hmm.c:hmm_range_fault_locked() {
	...
	if (hmm_select_migrate(range)) {
		WARN_ON_ONCE(range->pfn_flags_mask);
		WARN_ON_ONCE(range->dev_private_owner !=
			     range->migrate->pgmap_owner);
		memset(range->hmm_pfns, 0,
		       ((range->end - range->start) >> PAGE_SHIFT) *
		       sizeof(*range->hmm_pfns));
	}
	...
}

While the current address addr already ran hmm_vma_handle_pte() and keeps its
valid PFN, returning 0 directly from the PTE loop terminates processing for
the rest of the PMD range [addr + PAGE_SIZE, end) while reporting success to
walk_page_range().

Consequently, for any addresses in [addr + PAGE_SIZE, end), neither
hmm_vma_handle_pte() nor hmm_vma_handle_migrate_prepare() is called.
Independent migratable pages in that remainder are not collected, unmapped or
read-only PTEs are not faulted in even when HMM_PFN_REQ_FAULT is requested,
and hmm_range_fault() can return 0 with those range->hmm_pfns[] entries still
zeroed and lacking HMM_PFN_VALID.

[1] https://lore.kernel.org/r/20261021060506.2377545-5-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=4

Re: [PATCH v17 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/hmm: split of non-anonymous large folios in hmm_vma_handle_migrate_prepare()
- [High] mm/hmm: early PMD walk termination on folio split failure in hmm_vma_walk_pmd()
- [High] mm/hmm: reject KSM folios in hmm_vma_handle_migrate_prepare()
--

commit 9b85d918ada9ba7990835bcd807b9973c5976885
Author: Mika Penttilä <mpenttil@redhat.com>

mm/hmm: migrate collection in HMM pagewalk - pte level

This commit implements PTE-level migration preparation and collection in
hmm_vma_handle_migrate_prepare(), along with migrate_vma_split_folio() to
split PTE-mapped large folios during an HMM page table walk.

> diff --git a/mm/hmm.c b/mm/hmm.c
> index 854d2c090b20c..ec9280ef26822 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
> @@ -495,6 +495,71 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
>  #endif  /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
>  
>  #ifdef CONFIG_DEVICE_MIGRATION

[ ... ]

> @@ -512,8 +582,230 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
>  					  unsigned long *hmm_pfn,
>  					  bool *unmapped)
>  {
> -	// TODO: implement migration entry insertion
> +	struct hmm_vma_walk *hmm_vma_walk = walk->private;
> +	struct hmm_range *range = hmm_vma_walk->range;
> +	struct migrate_vma *migrate = range->migrate;

[ ... ]

> +		if (folio_test_large(folio)) {
> +			int ret;
> +
> +			ret = migrate_vma_split_folio(folio,
> +						      migrate->fault_page,
> +						      hmm_vma_walk,
> +						      ptep);
> +			if (ret)
> +				goto out_error;
> +			return -EAGAIN;
> +		}
> +
> +		pfn = page_to_pfn(page);
> +		if (softleaf_is_device_private_write(entry))
> +			writable = true;
> +	} else {
> +		pfn = pte_pfn(pte);
> +		if (is_zero_pfn(pfn) &&
> +		    (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
> +			/* Like pte_none(): only anonymous memory can be populated */
> +			if (!vma_is_anonymous(walk->vma))
> +				goto out;
> +			*hmm_pfn &= HMM_PFN_INOUT_FLAGS;
> +			*hmm_pfn |= HMM_PFN_MIGRATE;
> +			goto out;
> +		}
> +		page = vm_normal_page(walk->vma, addr, pte);
> +		if (page && !is_zone_device_page(page) &&
> +		    !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
> +			goto out;
> +		} else if (page && is_device_coherent_page(page)) {
> +			pgmap = page_pgmap(page);
> +
> +			if (!(minfo &
> +			      MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
> +			    pgmap->owner != migrate->pgmap_owner)
> +				goto out;
> +		}
> +
> +		folio = page ? page_folio(page) : NULL;
> +		if (folio && folio_test_large(folio)) {
> +			int ret;
> +
> +			ret = migrate_vma_split_folio(folio,
> +						      migrate->fault_page,
> +						      hmm_vma_walk,
> +						      ptep);
> +			if (ret)
> +				goto out_error;
> +			return -EAGAIN;
> +		}
> +
> +		writable = pte_write(pte);
> +	}

[Severity: High]
When walking a private file-backed VMA (which can contain a mix of
copy-on-write anonymous pages and read-only pagecache pages), will this
attempt to split non-anonymous large pagecache folios before checking
folio_test_anon()?

In the pte_present(pte) branch of hmm_vma_handle_migrate_prepare(),
migrate_vma_split_folio() is called on any large folio before verifying that
the folio is anonymous. If a PTE maps a read-only large file-backed folio in
the page cache, migrate_vma_split_folio() drops the PTE lock, takes the
sleeping folio_lock(folio) on the pagecache folio while already holding locks
on previously collected anonymous folios, and calls split_folio().

For file-backed folios, split_folio() -> __folio_freeze_split_file() takes
i_mmap_lock_read() and unmaps the folio without remapping it, leaving the
PTEs as pte_none() on the -EAGAIN retry, only for the resulting order-0
pagecache folio to be skipped by the !folio_test_anon() check. Furthermore,
if the filesystem sets mapping_min_folio_order() > 0, split_folio() fails
with -EINVAL (triggering VM_WARN_ONCE() in __folio_split()) and jumps to
out_error.

Should the !page || !folio_test_anon(folio) check happen before attempting to
split the folio in the present PTE path?

> +
> +	/* Only anonymous memory can be migrated to device memory */
> +	if (!page || !folio_test_anon(page_folio(page)))
> +		goto out;

[Severity: High]
Can KSM folios pass this check and be collected for device migration?

In include/linux/page-flags.h, FOLIO_MAPPING_KSM is defined as:

	FOLIO_MAPPING_ANON | FOLIO_MAPPING_ANON_KSM

Because FOLIO_MAPPING_ANON is set on KSM folios, folio_test_anon() returns
true for them. When hmm_range_fault() runs without HMM_PFN_REQ_WRITE (for
example during migrate_vma_setup() or a read fault with migration),
hmm_vma_handle_pte() does not break CoW on a read-only KSM PTE, and
hmm_vma_handle_migrate_prepare() locks and collects the KSM folio.

If the same KSM folio is mapped a second time later in the walked range,
hmm_vma_handle_pte() marks the second entry HMM_PFN_VALID before
folio_trylock() fails in hmm_vma_handle_migrate_prepare(), and
try_to_migrate() later unmaps that second mapping via rmap_walk_ksm(),
leaving a stale PFN in range->hmm_pfns. More importantly, migrating a KSM
folio to MEMORY_DEVICE_PRIVATE memory updates stable_node->kpfn via
folio_migrate_ksm() to a non-CPU-accessible ZONE_DEVICE PFN and installs
device-private swap entries into other processes sharing the KSM page,
causing ksmd to fault when it later accesses the page in memcmp_pages().

Should hmm_vma_handle_migrate_prepare() also reject folio_test_ksm(folio)?

[ ... ]

> +		set_pte_at(mm, addr, ptep, swp_pte);
> +		folio_remove_rmap_pte(folio, page, walk->vma);
> +		folio_put(folio);
> +		*hmm_pfn |= HMM_PFN_VALID | HMM_PFN_MIGRATE;
> +		if (pte_present(pte))
> +			*unmapped = true;
> +	} else {
> +		folio_put(folio);
> +	}
> +out:
>  	return 0;
> +out_error:
> +	return -EFAULT;
>  }

[Severity: High]
As reported by Sashiko on v16 [1], when migrate_vma_split_folio() fails in
hmm_vma_handle_migrate_prepare() and jumps to out_error (-EFAULT), does the
caller in hmm_vma_walk_pmd() abort the PMD walk too early and leave the
remaining pages in the PMD unprocessed?

In hmm_vma_walk_pmd(), the PTE loop handles non-zero returns from
hmm_vma_handle_migrate_prepare() by immediately returning 0:

	for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) {

		r = hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns);
		...
		r = hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &unmapped);
		if (r == -EAGAIN) {
			...
			goto again;
		}
		if (r) {
			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
			if (unmapped)
				flush_tlb_range(walk->vma, start, addr);
			return 0;
		}
	}

Because hmm_vma_handle_pte() is called inside the same per-page loop,
returning 0 from hmm_vma_walk_pmd() when a split fails at addr skips both
hmm_vma_handle_pte() and hmm_vma_handle_migrate_prepare() for all remaining
addresses in [addr + PAGE_SIZE, end) of the PMD. As a result,
hmm_range_fault() can return 0 (success) while range->hmm_pfns[] entries for
[addr + PAGE_SIZE, end) remain zeroed without HMM_PFN_VALID, and pages in
that tail of the PMD are neither faulted in (when HMM_PFN_REQ_FAULT is
requested) nor migrated.

[1] https://lore.kernel.org/r/20261021060506.2377545-6-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=5

Re: [PATCH v17 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] mm/hmm: missing folio_test_anon() check in hmm_vma_handle_migrate_prepare_pmd()
- [High] mm/hmm: missing device-coherent checks in hmm_vma_handle_migrate_prepare_pmd()
- [High] mm/hmm: ABBA folio lock deadlock in migrate_vma_split_folio()
- [Medium] mm/hmm: PMD split in hmm_vma_handle_migrate_prepare_pmd() aborts notifier walk
--

commit 2ed090a278a29feb7fac7d83140b748506dedc9e
Author: Mika Penttilä <mpenttil@redhat.com>

mm/hmm: migrate collection in HMM pagewalk - pmd level

This commit implements hmm_vma_handle_migrate_prepare_pmd() to collect
PMD-mapped THPs or split them into PTEs during HMM pagewalk migration, and
factors out __migrate_vma_split_folio() for splitting folios.

> diff --git a/mm/hmm.c b/mm/hmm.c
> index ec9280ef26822..cbde004fd778a 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
> @@ -511,9 +537,7 @@ static int migrate_vma_split_folio(struct folio *folio,
>  				   struct hmm_vma_walk *hmm_vma_walk,
>  				   pte_t *ptep)
>  {
> -	int ret;
>  	struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
> -	struct folio *new_fault_folio = NULL;
>  
>  	/*
>  	 * Take the reference while still holding the PTE lock, so the folio
> @@ -530,44 +554,136 @@ static int migrate_vma_split_folio(struct folio *folio,
>  	if (folio != fault_folio)
>  		folio_lock(folio);
>  
> -	ret = split_folio(folio);
> -	if (ret) {
> -		if (folio != fault_folio) {
> +	return __migrate_vma_split_folio(folio, fault_page);
> +}

[Severity: High]
Can calling folio_lock(folio) in migrate_vma_split_folio() cause an ABBA
deadlock when another folio is already held locked in the walk?

During a migrating hmm_range_fault() walk, the calling thread already holds
folio locks on fault_folio (when called from do_swap_page() ->
migrate_to_ram()) as well as on any folios collected at earlier addresses
in the range via hmm_vma_handle_migrate_prepare_pmd() or
hmm_vma_handle_migrate_prepare().

While normal collection in hmm_vma_handle_migrate_prepare() and
hmm_vma_handle_migrate_prepare_pmd() uses folio_trylock() to avoid deadlocking
against concurrent migrations or faults, migrate_vma_split_folio() drops the
PTE lock and sleeps unconditionally in folio_lock(folio):

  Thread 1                              Thread 2
  holds folio A locked
    (fault_folio or earlier entry)      holds folio B locked
  hmm_vma_handle_migrate_prepare()
    migrate_vma_split_folio(folio B)
      folio_lock(folio B)               migrate_vma_split_folio(folio A)
                                          folio_lock(folio A)
                                          (or migration_entry_wait(folio A))

Should migrate_vma_split_folio() also use folio_trylock(folio) and skip
migrating the folio if it cannot be locked?

> +
> +static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
> +					      pmd_t *pmdp,
> +					      unsigned long start,
> +					      unsigned long end,
> +					      unsigned long *hmm_pfn)
> +{
> +	struct hmm_vma_walk *hmm_vma_walk = walk->private;
> +	struct hmm_range *range = hmm_vma_walk->range;
> +	struct migrate_vma *migrate = range->migrate;
> +	struct folio *fault_folio = NULL;
> +	enum migrate_vma_info minfo;
> +	struct folio *folio;
> +	unsigned long i;
> +	int r = 0;
> +
> +	// Do we want to migrate at all?
> +	minfo = hmm_select_migrate(range);
> +	if (!minfo)
> +		return r;
> +
> +	WARN_ON_ONCE(!migrate);
> +	HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true);
> +
> +	fault_folio = migrate->fault_page ?
> +		page_folio(migrate->fault_page) : NULL;
> +
> +	if (pmd_none(*pmdp))
> +		return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
> +
> +	/* Like at the pte level, NUMA hinting (protnone) pmds are migrated */
> +	if (!(hmm_pfn[0] & HMM_PFN_VALID) &&
> +	    !(pmd_trans_huge(*pmdp) && pmd_protnone(*pmdp)))
> +		goto out;
> +
> +	if (pmd_trans_huge(*pmdp)) {
> +		if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM))
> +			goto out;

[Severity: High]
How are PMD-mapped MEMORY_DEVICE_COHERENT folios handled here?

Compound device-coherent folios are mapped as present huge PMDs where
pmd_trans_huge(*pmdp) is true (via migrate_vma_insert_huge_pmd_page() and
remove_migration_pmd()). At the PTE level, hmm_vma_handle_migrate_prepare()
distinguishes system memory from device-coherent pages:

mm/hmm.c:hmm_vma_handle_migrate_prepare() {
    ...
		page = vm_normal_page(walk->vma, addr, pte);
		if (page && !is_zone_device_page(page) &&
		    !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
			goto out;
		} else if (page && is_device_coherent_page(page)) {
			pgmap = page_pgmap(page);

			if (!(minfo &
			      MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
			    pgmap->owner != migrate->pgmap_owner)
				goto out;
		}
    ...
}

In hmm_vma_handle_migrate_prepare_pmd(), requiring MIGRATE_VMA_SELECT_SYSTEM
for all pmd_trans_huge(*pmdp) entries without checking
folio_is_zone_device(folio), folio_is_device_coherent(folio),
MIGRATE_VMA_SELECT_DEVICE_COHERENT, or folio->pgmap->owner means that
migrating device-coherent memory to system RAM with
MIGRATE_VMA_SELECT_DEVICE_COHERENT skips present device-coherent huge PMDs,
while migrating with MIGRATE_VMA_SELECT_SYSTEM collects or splits
device-coherent huge PMDs regardless of pgmap->owner.

Does the pmd_trans_huge(*pmdp) branch need the same zone_device and
device_coherent checks as hmm_vma_handle_migrate_prepare()?

> +
> +		folio = pmd_folio(*pmdp);
> +		if (is_huge_zero_folio(folio))
> +			return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
> +
> +	} else if (!pmd_present(*pmdp)) {

[Severity: High]
Should hmm_vma_handle_migrate_prepare_pmd() check folio_test_anon(folio)
before collecting or splitting a PMD-mapped folio?

At the PTE level, hmm_vma_handle_migrate_prepare() rejects non-anonymous
folios:

mm/hmm.c:hmm_vma_handle_migrate_prepare() {
    ...
	/* Only anonymous memory can be migrated to device memory */
	if (!page || !folio_test_anon(page_folio(page)))
		goto out;
    ...
}

Because hmm_vma_capture_migrate_range() and migrate_vma_setup() allow file and
shmem VMAs, a PMD in such a VMA can map a non-anonymous THP. Without a
folio_test_anon(folio) check in hmm_vma_handle_migrate_prepare_pmd():

1. If compound migration is selected, the page-cache folio is locked and a PMD
   migration entry is installed with HMM_PFN_MIGRATE | HMM_PFN_COMPOUND.
   Holding a page-cache folio locked while hmm_do_fault() -> handle_mm_fault()
   faults subsequent pages in the VMA inverts the mapping->invalidate_lock ->
   folio_lock ordering. If the caller later splits the unmapped compound
   folio in __migrate_device_pages() -> migrate_vma_split_unmapped_folio(),
   split_huge_pmd_address() zaps the non-anonymous PMD without dropping the
   extra folio reference, and folio_split_unmapped() triggers
   VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio) before running
   __folio_freeze_split_anon() on a page-cache folio:

mm/huge_memory.c:folio_split_unmapped() {
	VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
	VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
	VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);

	return __folio_freeze_split_anon(folio, new_order, &folio->page,
					 false, NULL, SPLIT_TYPE_UNIFORM);
}

2. If the split fallback is taken, __migrate_vma_split_folio() ->
   split_folio() -> __folio_freeze_split_file() acquires
   i_mmap_lock_read(mapping) while holding locks on previously collected
   folios and unmaps the file folio completely via unmap_folio(). If the
   split fails after unmapping, hmm_vma_handle_migrate_prepare_pmd() returns 0
   with stale HMM_PFN_VALID entries for a zapped PMD; if it succeeds, the
   re-walk finds pte_none entries and loses the valid PFNs when
   HMM_PFN_REQ_FAULT is not set.

[ ... ]

> +split:
> +	spin_unlock(hmm_vma_walk->ptl);
> +	hmm_vma_walk->pmdlocked = false;
>  
>  	/*
> -	 * Ensure the lock is held on the correct
> -	 * folio after the split
> +	 * folio_get() above took an extra reference. For the fault folio the
> +	 * caller still holds a reference and the lock, which is the
> +	 * precondition of __migrate_vma_split_folio(), so drop the extra one.
>  	 */
> -	if (!new_fault_folio) {
> -		folio_unlock(folio);
> +	if (folio == fault_folio)
>  		folio_put(folio);
> -	} else if (folio != new_fault_folio) {
> -		if (new_fault_folio != fault_folio) {
> -			folio_get(new_fault_folio);
> -			folio_lock(new_fault_folio);
> -		}
> -		folio_unlock(folio);
> -		folio_put(folio);
> -	}
>  
> -	return 0;
> -}
> +	/* Split failed: not migrated, the valid pfns stay as reported */
> +	if (__migrate_vma_split_folio(folio, migrate->fault_page))
> +		return 0;
>  
> -static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
> -					      pmd_t *pmdp,
> -					      unsigned long start,
> -					      unsigned long end,
> -					      unsigned long *hmm_pfn)
> -{
> -	// TODO: implement migration entry insertion
> -	return 0;
> +	/* Re-walked as PTEs from start: drop the pmd-level output */
> +	for (i = 0; i < (end - start) >> PAGE_SHIFT; i++)
> +		hmm_pfn[i] &= HMM_PFN_INOUT_FLAGS;
> +
> +	hmm_vma_walk->last = start;
> +	return -EBUSY;
>  }
>  
>  /*
> @@ -1016,6 +1132,11 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
>  			hmm_vma_walk->pmdlocked = false;
>  		}
>  
> +		/*
> +		 * hmm_vma_handle_migrate_prepare_pmd() splits the huge pmd in
> +		 * place when needed and returns -EBUSY to re-walk the range as
> +		 * PTEs.
> +		 */
>  		return r;
>  	}

[Severity: Medium]
When range->notifier is set during a migrating hmm_range_fault() call, does
returning -EBUSY out of walk_page_range() after splitting a PMD cause
hmm_range_fault_locked() to abort the walk instead of re-walking the split
PMD as PTEs?

In hmm_vma_handle_migrate_prepare_pmd(), __migrate_vma_split_folio() calls
split_folio() or split_huge_page() -> __folio_split() ->
__folio_freeze_split_anon() -> unmap_folio() -> try_to_migrate_one(), which
invokes mmu_notifier_invalidate_range_start() with MMU_NOTIFY_CLEAR and
range.owner == NULL. Because the event is MMU_NOTIFY_CLEAR rather than
MMU_NOTIFY_MIGRATE with range->dev_private_owner, the registered
mmu_interval_notifier updates range->notifier->invalidate_seq.

When hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_walk_pmd() return
-EBUSY back to hmm_range_fault_locked(), the next iteration of the loop checks
mmu_interval_check_retry():

mm/hmm.c:hmm_range_fault_locked() {
    ...
	do {
		/* If range is no longer valid force retry. */
		if (range->notifier && mmu_interval_check_retry(range->notifier,
								range->notifier_seq)) {
			ret =  -EBUSY;
			break;
		}
		ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
				      &hmm_walk_ops, &hmm_vma_walk);
    ...
}

Because range->notifier was invalidated by the split,
mmu_interval_check_retry() returns true and hmm_range_fault_locked() breaks
out with -EBUSY, forcing a full rollback and retry of the range. By contrast,
the PTE-level split in hmm_vma_handle_migrate_prepare() returns -EAGAIN and
restarts locally at the again label inside hmm_vma_walk_pmd() without
returning to hmm_range_fault_locked().

Could hmm_vma_walk_pmd() similarly loop locally to the again label when
hmm_vma_handle_migrate_prepare_pmd() splits the PMD?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=6

Re: [PATCH v17 08/12] mm/hmm: implement rollback for device page migration in HMM pagewalk

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/hmm: missing rollback of prior entries when restarting after fault
- [High] mm/hmm: missing rollback and incomplete walk on folio split failure
- [High] mm/hmm: self-deadlock on -EBUSY retry due to memset() of hmm_pfns
--

commit 57b87c4290e06aea8db775c69fa86c746acfe43d
Author: Mika Penttilä <mpenttil@redhat.com>

mm/hmm: implement rollback for device page migration in HMM pagewalk

This commit adds rollback handling to unlock and unreference already
collected folios if page tables vanish or change while page table locks are
dropped during a migration pagewalk. It also skips migration PTEs installed
earlier in the same walk when restarting a PMD walk.

> diff --git a/mm/hmm.c b/mm/hmm.c
> index 690b73908f34e..1c8a3fa29c4e6 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c

[ ... ]

> @@ -331,6 +344,15 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
>  	if (!pte_present(pte)) {
>  		const softleaf_t entry = softleaf_from_pte(pte);
>  
> +		/*
> +		 * Our own migration entry, collected earlier in this walk:
> +		 * keep the collected pfn, don't fault or roll it back.
> +		 */
> +		if (softleaf_is_migration(entry) && hmm_select_migrate(range) &&
> +		    hmm_pfn_collected(*hmm_pfn) &&
> +		    hmm_pfn_to_page(*hmm_pfn) == softleaf_to_page(entry))
> +			return 0;

[Severity: High]
When hmm_range_fault_locked() aborts with -EBUSY (or a caller retries
hmm_range_fault() after mmu_interval_read_retry() trips) after collecting
earlier pages in the range, those folios remain locked, referenced, and
mapped as migration PTEs.

When the caller retries hmm_range_fault() with the same range and hmm_pfns
array, hmm_range_fault_locked() zeroes range->hmm_pfns:

mm/hmm.c:hmm_range_fault_locked() {
    ...
	if (hmm_select_migrate(range)) {
		WARN_ON_ONCE(range->pfn_flags_mask);
		WARN_ON_ONCE(range->dev_private_owner !=
			     range->migrate->pgmap_owner);
		memset(range->hmm_pfns, 0,
		       ((range->end - range->start) >> PAGE_SHIFT) *
		       sizeof(*range->hmm_pfns));
	}
    ...
}

On the retry walk, hmm_pfn_collected(*hmm_pfn) evaluates to false for the
migration PTEs installed by the current thread on the previous attempt. Does
this cause hmm_vma_handle_pte() and hmm_vma_walk_pmd() to treat our own
migration entries as belonging to another thread and call
hmm_do_fault()->handle_mm_fault()->migration_entry_wait() on folios already
locked by current, resulting in a self-deadlock?

> +
>  		/*
>  		 * Don't fault in device private pages owned by the caller,
>  		 * just report the PFN.
> @@ -416,6 +438,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
>  
>  	new_pfn_flags = pte_pfn(pte) | cpu_flags;
>  out:
> +	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
> +						addr + PAGE_SIZE,
> +						hmm_pfn);
>  	*hmm_pfn = (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags;
>  	return 0;
>  
> @@ -427,7 +452,19 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
>  	} else {
>  		pte_unmap(ptep);
>  	}
> -	/* Fault any virtual address we were asked to fault */
> +	/*
> +	 * This pte needs a fault, so it is not our migration entry: a pfn
> +	 * collected here earlier is gone, roll it back before faulting.
> +	 */
> +	hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
> +						addr + PAGE_SIZE, hmm_pfn);
> +	/*
> +	 * Fault any virtual address we were asked to fault. When migrating,
> +	 * only this page: later ptes may be our own migration entries or
> +	 * caller owned device private pages, which must not be faulted.
> +	 */
> +	if (hmm_select_migrate(range))
> +		end = addr + PAGE_SIZE;
>  	return hmm_record_fault(addr, end, required_fault, walk);
>  }

[Severity: High]
As reported by Sashiko on v16 [1], when a PTE fault occurs at addr after
earlier PTEs at addresses below addr have already been collected by
hmm_vma_handle_migrate_prepare(), the fault path drops hmm_vma_walk->ptl,
rolls back only [addr, addr + PAGE_SIZE), and calls hmm_record_fault(),
which sets hmm_vma_walk->last = addr and returns HMM_FAULT_PENDING.

In hmm_range_fault_locked(), HMM_FAULT_PENDING disarms the migration
invalidation range via hmm_vma_migrate_disarm() and runs hmm_do_fault()
without holding any page table spinlock, then resumes walk_page_range() at
hmm_vma_walk.last:

mm/hmm.c:hmm_range_fault_locked() {
    ...
		ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
				      &hmm_walk_ops, &hmm_vma_walk);
		if (ret == HMM_FAULT_PENDING) {
			hmm_vma_migrate_disarm(&hmm_vma_walk);
			ret = hmm_do_fault(mm, &hmm_vma_walk);
    ...
}

Because walk_page_range() restarts at addr rather than range->start,
hmm_vma_walk_pmd() is invoked with start = addr (and any earlier PMDs in
[range->start, addr) are skipped completely).

If the migration PTEs or PTE page table covering [range->start, addr) were
zapped or replaced while locks were dropped during hmm_do_fault(), can those
earlier collected entries in range->hmm_pfns ever be revisited or rolled
back, or will they leak folio locks and references?

[ ... ]

> @@ -1192,6 +1311,12 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
>  				flush_tlb_range(walk->vma, start, addr);
>  				unmapped = false;
>  			}
> +			/*
> +			 * The pte lock was dropped and the pte table may have
> +			 * changed: re-walk from start. Our own migration entries
> +			 * are kept, anything else collected is rolled back.
> +			 */
> +			addr = start;
>  			goto again;
>  		}
>  		if (r) {

[Severity: High]
When hmm_vma_handle_migrate_prepare() attempts to split a large folio via
migrate_vma_split_folio(), it unconditionally drops hmm_vma_walk->ptl and
sleeps on folio_lock() before calling __migrate_vma_split_folio():

mm/hmm.c:migrate_vma_split_folio() {
    ...
	lazy_mmu_mode_disable();
	pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
	hmm_vma_walk->ptelocked = false;

	if (folio != fault_folio)
		folio_lock(folio);

	return __migrate_vma_split_folio(folio, fault_page);
}

If __migrate_vma_split_folio() fails, hmm_vma_handle_migrate_prepare()
returns -EFAULT, and the if (r) branch in hmm_vma_walk_pmd() flushes the TLB
and returns 0 immediately:

mm/hmm.c:hmm_vma_walk_pmd() {
    ...
		if (r) {
			/*
			 * Splitting the folio failed; migrate_vma_split_folio()
			 * already dropped the PTE lock. Don't migrate the rest
			 * of the pmd range; addr keeps its valid pfn.
			 */
			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
			if (unmapped)
				flush_tlb_range(walk->vma, start, addr);
			return 0;
		}
    ...
}

Since the PTE lock was dropped while waiting on folio_lock(), if migration
PTEs or the PTE table in [start, end) vanished during that window, won't
returning 0 without re-walking [start, end) skip rolling back those
collected folios, and also leave range->hmm_pfns unpopulated for the
remaining addresses in [addr + PAGE_SIZE, end)?

[1] https://lore.kernel.org/r/20261021060506.2377545-9-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=8

Re: [PATCH v17 09/12] mm: enable device page migration from HMM pagewalk

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] mm: hmm_range_fault_locked() zeroes hmm_pfns on -EBUSY migration retry
- [Critical] mm/hmm: missing rollback of earlier entries after fault in hmm_range_fault()
- [High] mm: missing folio_test_anon() check in hmm_vma_handle_migrate_prepare_pmd()
- [High] mm: early return in hmm_vma_walk_pmd() on folio split failure skips PTEs
- [High] mm/hmm: folio_lock deadlock in migrate_vma_split_folio() and hmm_do_fault()
- [Medium] mm: lost MIGRATE_PFN_WRITE on writable protnone entries in HMM migration
- [Low] mm: build failure on incomplete struct migrate_vma when !CONFIG_MIGRATION
- [Low] mm/migrate_device: unused MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE flags in migrate_vma_setup()
--

commit 00843344cdcc1e298091d82f4ddd5b2d3478ddb1
Author: Mika Penttilä <mpenttil@redhat.com>

mm: enable device page migration from HMM pagewalk

This commit enables migration collection in hmm_range_fault() via
hmm_select_migrate(), switches migrate_vma_setup() to use the HMM pagewalk
path with new MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE flags, and removes
the old migrate_vma_collect() page table walk helpers.

> diff --git a/include/linux/migrate.h b/include/linux/migrate.h
> index dd7dd7ec6995e..3780c4c116b8a 100644
> --- a/include/linux/migrate.h
> +++ b/include/linux/migrate.h
> @@ -181,6 +181,8 @@ enum migrate_vma_info {
>  	MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
>  	MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
>  	MIGRATE_VMA_SELECT_COMPOUND = 1 << 3,
> +	MIGRATE_VMA_FAULT = 1 << 4,
> +	MIGRATE_VMA_WRITE = 1 << 5,
>  };

[Severity: Low]
As reported by Christoph Hellwig on v15 [1], are MIGRATE_VMA_FAULT and
MIGRATE_VMA_WRITE in enum migrate_vma_info and their handling in
migrate_vma_setup() needed when there are no in-tree callers?

Later patches in the series (for test_hmm and drm/nouveau/svm) drive
migrate-on-fault directly through hmm_range_fault() with HMM_PFN_REQ_MIGRATE
and migrate_hmm_range_setup(), leaving MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE
unused across the tree.

> @@ -218,10 +220,14 @@ struct migrate_vma {
>  	struct page		*fault_page;
>  };
>  
> -// TODO: enable migration
>  static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
>  {
> -	return 0;
> +	enum migrate_vma_info minfo;
> +
> +	minfo = (range->default_flags & HMM_PFN_REQ_MIGRATE) ?
> +		range->migrate->flags : 0;
> +
> +	return minfo;
>  }

[Severity: Low]
Will mm/hmm.c fail to build when CONFIG_HMM_MIRROR=y and CONFIG_MIGRATION=n?

Several drivers (such as INFINIBAND_ON_DEMAND_PAGING, DRM_AMDGPU_USERPTR, and
MSHV_ROOT) select CONFIG_HMM_MIRROR without selecting CONFIG_MIGRATION. When
CONFIG_MIGRATION is disabled, struct migrate_vma is only forward-declared in
include/linux/hmm.h, but hmm_range_fault_locked() in mm/hmm.c is compiled
outside #ifdef CONFIG_DEVICE_MIGRATION and dereferences
range->migrate->pgmap_owner:

mm/hmm.c:hmm_range_fault_locked() {
	if (hmm_select_migrate(range)) {
		WARN_ON_ONCE(range->pfn_flags_mask);
		WARN_ON_ONCE(range->dev_private_owner !=
			     range->migrate->pgmap_owner);
		...
	}
}

In addition, hmm_select_migrate() in include/linux/migrate.h is guarded by
#ifdef CONFIG_MIGRATION rather than #ifdef CONFIG_DEVICE_MIGRATION, even
though the migration implementation in mm/hmm.c and mm/migrate_device.c
depends on CONFIG_DEVICE_MIGRATION.

Should hmm_select_migrate() be guarded by CONFIG_DEVICE_MIGRATION (returning 0
when !CONFIG_DEVICE_MIGRATION), and should the range->migrate dereference in
hmm_range_fault_locked() be guarded or moved into a helper?

[ ... ]

> diff --git a/mm/hmm.c b/mm/hmm.c
> index 1c8a3fa29c4e6..0b38bf84c3690 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
> @@ -1658,14 +1658,43 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
>   *		the invalidation to finish.
>   * -EFAULT:     A page was requested to be valid and could not be made valid
>   *              ie it has no backing VMA or it is illegal to access
> + * -ERANGE:     The range crosses multiple VMAs, or space for hmm_pfns array
> + *              is too low.
>   *
>   * This is similar to get_user_pages(), except that it can read the page tables
>   * without mutating them (ie causing faults).
>   *
>   * The mmap lock must be held by the caller and will remain held on return.
>   * New users should prefer hmm_range_fault_unlocked_timeout() unless they
> - * specifically need to keep the mmap lock held across the call. This helper
> - * cannot support VMAs whose fault handlers need to drop the mmap lock.
> + * specifically need to keep the mmap lock held across the call like while
> + * migrating. This helper cannot support VMAs whose fault handlers need to
> + * drop the mmap lock.
> + *
> + * If want to do migration after faulting, call hmm_range_fault() with
> + * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally
> + * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate field.
> + * range->migrate->vma will be populated during the call,
> + * and must be stable across the whole migrate process, which is
> + * why mmap_lock must be held around this call.
> + *
> + * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on
> + * return and the caller takes responsibility for completing or aborting it.
> + *
> + * On success, the caller must call migrate_hmm_range_setup() and may then
> + * proceed with the normal migrate_vma sequence: prepare destination pages,
> + * call migrate_vma_pages(), update device mappings as needed, and finally
> + * call migrate_vma_finalize().
> + *
> + * On -EBUSY, the caller may retry hmm_range_fault() using the same range and
> + * PFN array without undoing entries collected by the previous attempt. If
> + * the caller stops retrying, it must abort the partial migration.

[Severity: Critical]
When a caller retries hmm_range_fault() after -EBUSY with the same range and
PFN array as documented here, won't hmm_range_fault_locked() wipe out all
previously collected entries by unconditionally zeroing range->hmm_pfns?

At the start of hmm_range_fault_locked() in mm/hmm.c:

	if (hmm_select_migrate(range)) {
		WARN_ON_ONCE(range->pfn_flags_mask);
		WARN_ON_ONCE(range->dev_private_owner !=
			     range->migrate->pgmap_owner);
		memset(range->hmm_pfns, 0,
		       ((range->end - range->start) >> PAGE_SHIFT) *
		       sizeof(*range->hmm_pfns));
	}

During the first call, hmm_vma_handle_migrate_prepare() and
hmm_vma_handle_migrate_prepare_pmd() lock folios, take references, install
migration PTEs/PMDs, and record HMM_PFN_VALID | HMM_PFN_MIGRATE in
range->hmm_pfns[]. If a later page in the range triggers an mmu_notifier
invalidation (for example during hmm_do_fault() or folio splitting) so that
mmu_interval_check_retry() returns -EBUSY, retrying hmm_range_fault() zeroes
range->hmm_pfns[].

On the retry walk, hmm_pfn_collected() in hmm_vma_handle_pte() and
hmm_vma_walk_pmd() evaluates to false for the migration entries installed by
the first attempt:

mm/hmm.c:hmm_vma_handle_pte() {
	if (softleaf_is_migration(entry) && hmm_select_migrate(range) &&
	    hmm_pfn_collected(*hmm_pfn) &&
	    hmm_pfn_to_page(*hmm_pfn) == softleaf_to_page(entry))
		return 0;
}

With HMM_PFN_REQ_FAULT set, HMM treats its own migration PTE/PMD as a foreign
migration entry and calls hmm_do_fault() -> handle_mm_fault() ->
migration_entry_wait(), deadlocking on the folio lock already held by current.
Without HMM_PFN_REQ_FAULT (or if the retry aborts before reaching those
entries), range->hmm_pfns[] remains zeroed, so migrate_hmm_range_setup() never
populates migrate->src[] for those pages and leaks the folio locks, folio
references, and migration PTEs.

[ ... ]

> diff --git a/mm/migrate_device.c b/mm/migrate_device.c
> index ba6775a0d37e6..266f9c6d7b607 100644
> --- a/mm/migrate_device.c
> +++ b/mm/migrate_device.c

[ ... ]

> @@ -741,9 +243,20 @@ int migrate_vma_setup(struct migrate_vma *args)
>  {
>  	const struct vm_area_struct *vma = args->vma;
>  	long nr_pages = (args->end - args->start) >> PAGE_SHIFT;
> +	struct hmm_range range = {
> +		.notifier = NULL,
> +		.hmm_pfns = args->src,
> +		.dev_private_owner = args->pgmap_owner,
> +		.migrate = args,
> +		.default_flags = HMM_PFN_REQ_MIGRATE
> +	};
> +	int ret;
>  
>  	args->start &= PAGE_MASK;
>  	args->end &= PAGE_MASK;
> +	range.start = args->start;
> +	range.end = args->end;
> +
>  	if (!vma)
>  		return -EINVAL;
>  	/* The migrate_vma page walk has no hugetlb support. */
> @@ -766,10 +279,24 @@ int migrate_vma_setup(struct migrate_vma *args)
>  	args->cpages = 0;
>  	args->npages = 0;
>  
> -	migrate_vma_collect(args);
> +	if (args->flags & MIGRATE_VMA_FAULT)
> +		range.default_flags |= HMM_PFN_REQ_FAULT;
>  
> -	if (args->cpages)
> -		migrate_vma_unmap(args);
> +	if (args->flags & MIGRATE_VMA_WRITE)
> +		range.default_flags |= HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE;
> +
> +	ret = hmm_range_fault(&range);
> +
> +	migrate_hmm_range_setup(&range);

[Severity: Critical]
As reported by Sashiko on v16 [2], when hmm_range_fault() faults at an address
after already collecting earlier PTEs in the range, can concurrent page table
zapping cause a use-after-free of destination pages because the walk resumes
at hmm_vma_walk.last without re-validating [range->start, hmm_vma_walk.last)?

When HMM_PFN_REQ_MIGRATE | HMM_PFN_REQ_FAULT is used, walk_page_range()
collects present pages in [range->start, addr) via
hmm_vma_handle_migrate_prepare(), installing migration PTEs and setting
HMM_PFN_VALID | HMM_PFN_MIGRATE in range->hmm_pfns[]. When a later address
addr requires a fault, hmm_vma_handle_pte() drops the PTE lock, sets
hmm_vma_walk.last = addr via hmm_record_fault(), and returns
HMM_FAULT_PENDING:

mm/hmm.c:hmm_range_fault_locked() {
	ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
			      &hmm_walk_ops, &hmm_vma_walk);
	if (ret == HMM_FAULT_PENDING) {
		hmm_vma_migrate_disarm(&hmm_vma_walk);
		ret = hmm_do_fault(mm, &hmm_vma_walk);
		...
	}
}

While the PTE spinlock and migration mmu_notifier invalidation are dropped
during hmm_do_fault(), a concurrent madvise(MADV_DONTNEED) only requires
mmap_read_lock and can clear the migration PTEs in [range->start, addr) via
zap_pte_range() -> zap_nonpresent_ptes() without taking the folio lock. When
the loop resumes walk_page_range() at hmm_vma_walk.last = addr,
hmm_vma_handle_migrate_prepare_rollback() only inspects [addr, end) and never
re-checks [range->start, addr).

Those zapped entries keep HMM_PFN_VALID | HMM_PFN_MIGRATE in range->hmm_pfns[]
and pass migrate_vma_check_page() because folio_mapped(src) is false. Later,
in __migrate_device_finalize(), remove_migration_ptes(src, dst, 0) finds no
migration PTEs to replace (so dst is never mapped into the CPU page tables or
given a page table refcount), and folio_put(dst) frees the destination folio
while the device driver still has it mapped.

[Severity: High]
As reported by Sashiko on v16 [3], can switching migrate_vma_setup() and
hmm_range_fault() to the HMM migration path deadlock on folio locks in
migrate_vma_split_folio() and hmm_do_fault(), or cause spurious split_folio()
-EAGAIN failures?

First, during migration collection, hmm_vma_handle_migrate_prepare() keeps all
previously collected folios in the range locked (using folio_trylock() to
avoid deadlocks). However, when it encounters a PTE-mapped large folio, it
calls migrate_vma_split_folio(), which takes a reference and does a blocking
folio_lock() while still holding the locks and migration PTEs on fault_folio
and all earlier collected folios:

mm/hmm.c:migrate_vma_split_folio() {
	if (folio != fault_folio)
		folio_get(folio);

	lazy_mmu_mode_disable();
	pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
	hmm_vma_walk->ptelocked = false;

	if (folio != fault_folio)
		folio_lock(folio);

	return __migrate_vma_split_folio(folio, fault_page);
}

If another task holds folio_lock(folio) and waits on one of the earlier
migration PTEs via migration_entry_wait() (or tries to lock one of the
already-collected folios), both tasks deadlock. Taking folio_get(folio) before
folio_lock(folio) also means a concurrent caller blocking on folio_lock(folio)
elevates the refcount and causes the lock holder's split_folio() to fail with
-EAGAIN.

Second, when HMM_PFN_REQ_FAULT is used with HMM_PFN_REQ_MIGRATE, encountering
another thread's migration entry in hmm_vma_handle_pte() or hmm_vma_walk_pmd()
(or a swap or device-private entry) invokes hmm_do_fault() ->
handle_mm_fault(), which sleeps in migration_entry_wait() or folio_lock()
while still holding the locks and migration PTEs on all folios collected
earlier in the walk.

[Severity: High]
Does hmm_vma_handle_migrate_prepare_pmd() in mm/hmm.c miss a
folio_test_anon(folio) check when collecting or splitting a PMD-mapped THP?

At the PTE level, hmm_vma_handle_migrate_prepare() rejects non-anonymous
folios:

mm/hmm.c:hmm_vma_handle_migrate_prepare() {
	/* Only anonymous memory can be migrated to device memory */
	if (!page || !folio_test_anon(page_folio(page)))
		goto out;
}

However, hmm_vma_handle_migrate_prepare_pmd() selects any non-zero
pmd_trans_huge(*pmdp) folio when MIGRATE_VMA_SELECT_SYSTEM is set:

mm/hmm.c:hmm_vma_handle_migrate_prepare_pmd() {
	if (pmd_trans_huge(*pmdp)) {
		if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM))
			goto out;

		folio = pmd_folio(*pmdp);
		if (is_huge_zero_folio(folio))
			return hmm_pfns_fill(start, end, hmm_vma_walk, 0);

	}
}

If HMM migration runs on a private file-backed or shmem VMA containing a
PMD-mapped page-cache/shmem THP, hmm_vma_handle_migrate_prepare_pmd() locks
the non-anonymous folio and either installs a PMD migration entry (holding the
file/shmem folio lock across subsequent faults in hmm_do_fault(), which take
mapping->invalidate_lock and invert the mapping->invalidate_lock -> folio_lock
ordering) or splits the THP even though hmm_vma_handle_migrate_prepare() will
reject all of its PTEs.

[Severity: High]
In hmm_vma_walk_pmd() in mm/hmm.c, when splitting a PTE-mapped large folio
fails in hmm_vma_handle_migrate_prepare(), doesn't returning 0 immediately
skip the remaining PTEs in [addr + PAGE_SIZE, end)?

In the PTE loop of hmm_vma_walk_pmd():

mm/hmm.c:hmm_vma_walk_pmd() {
	for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) {
		...
		r = hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns);
		...
		r = hmm_vma_handle_migrate_prepare(walk, pmdp, addr, ptep,
						   hmm_pfns, &unmapped);
		if (r == -EAGAIN) {
			addr = start;
			goto again;
		}
		if (r) {
			HMM_ASSERT_UNLOCKED(hmm_vma_walk);
			if (unmapped)
				flush_tlb_range(walk->vma, start, addr);
			return 0;
		}
	}
}

When migrate_vma_split_folio() fails at addr, it drops the PTE lock and
hmm_vma_handle_migrate_prepare() returns -EFAULT. Returning 0 from
hmm_vma_walk_pmd() tells walk_page_range() that the entire PMD range
[start, end) is complete, leaving [addr + PAGE_SIZE, end) unvisited and
unpopulated in range->hmm_pfns[]. Moreover, if an earlier pass in
hmm_vma_walk_pmd() had already collected PTEs at addresses above addr before
restarting via addr = start; goto again, dropping the PTE lock at addr and
returning 0 skips re-verifying and rolling back those collected entries in
[addr + PAGE_SIZE, end).

[Severity: Medium]
Can writable protnone PTEs and PMDs lose MIGRATE_PFN_WRITE in migrate->src[]
when migrated via migrate_vma_setup() or hmm_range_fault() without
HMM_PFN_REQ_FAULT?

For a present protnone PTE or PMD, pte_to_hmm_pfn_flags() and
pmd_to_hmm_pfn_flags() return 0, so HMM_PFN_WRITE is not set in *hmm_pfn.
Then hmm_vma_handle_migrate_prepare() and hmm_vma_handle_migrate_prepare_pmd()
collect the protnone entry, install a writable migration entry if
pte_write(pte) or pmd_write(*pmdp) is true, and set HMM_PFN_VALID |
HMM_PFN_MIGRATE in *hmm_pfn, but never set HMM_PFN_WRITE:

mm/hmm.c:hmm_vma_handle_migrate_prepare() {
		set_pte_at(mm, addr, ptep, swp_pte);
		folio_remove_rmap_pte(folio, page, walk->vma);
		folio_put(folio);
		*hmm_pfn |= HMM_PFN_VALID | HMM_PFN_MIGRATE;
}

When migrate_hmm_range_setup() builds migrate->src[i] from range->hmm_pfns[i]:

mm/migrate_device.c:migrate_hmm_range_setup() {
		migrate->src[i] |= MIGRATE_PFN_MIGRATE;
		migrate->src[i] |= (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0;
		migrate->src[i] |= (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0;
}

MIGRATE_PFN_WRITE is left unset even though the deleted
migrate_vma_collect_pmd() and migrate_vma_collect_huge_pmd() helpers preserved
MIGRATE_PFN_WRITE whenever pte_write(pte) or pmd_write(*pmdp) was set. Should
hmm_vma_handle_migrate_prepare() and hmm_vma_handle_migrate_prepare_pmd() also
set HMM_PFN_WRITE on *hmm_pfn when writable is true?

[1] https://lore.kernel.org/r/arY5zKsLuf7f_EYw@infradead.org
[2] https://lore.kernel.org/r/20261021060506.2377545-10-mpenttil@redhat.com
[3] https://lore.kernel.org/r/20261021060506.2377545-10-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=9

Re: [PATCH v17 10/12] lib/test_hmm: add a new testcase for the migrate on fault

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm: test_hmm: deadlock and folio leak on retry in dmirror_range_fault()
- [Medium] mm: test_hmm: unmapped device pages on retry in do_fault_and_migrate()
- [Low] lib/test_hmm: excessive stack frame in dmirror_fault_and_migrate_to_device()
--

commit 7670468c9cdf50d8e222db90061981491fa308c8
Author: Mika Penttilä <mpenttil@redhat.com>

lib/test_hmm: add a new testcase for the migrate on fault

This commit adds a migrate-on-fault testcase to the HMM test driver and
selftests using HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE with
hmm_range_fault() to fault in and migrate ranges to device-private memory.

> diff --git a/lib/test_hmm.c b/lib/test_hmm.c
> index f05f801b8404b..2045896493ae3 100644
> --- a/lib/test_hmm.c
> +++ b/lib/test_hmm.c
> @@ -36,6 +36,7 @@
>  #define DMIRROR_RANGE_FAULT_TIMEOUT	1000
>  #define DEVMEM_CHUNK_SIZE		(256 * 1024 * 1024U)
>  #define DEVMEM_CHUNKS_RESERVE		16
> +#define PFNS_ARRAY_SIZE			64
>  
>  /*
>   * For device_private pages, dpage is just a dummy struct page
> @@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror,
>  	struct mm_struct *mm = dmirror->notifier.mm;
>  	unsigned long timeout =
>  		jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
> +	bool migrate = range->default_flags & HMM_PFN_REQ_MIGRATE;
>  	int ret;
>  
>  	while (true) {
> @@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror,
>  		}
>  
>  		range->notifier_seq = mmu_interval_read_begin(range->notifier);
> -		mmap_read_lock(mm);
> -		ret = hmm_range_fault(range);
> -		mmap_read_unlock(mm);
> +
> +		/* mmap lock held for whole migrate on fault operation */
> +		if (!migrate) {
> +			mmap_read_lock(mm);
> +			ret = hmm_range_fault(range);
> +			mmap_read_unlock(mm);
> +		} else {
> +			ret = hmm_range_fault(range);
> +		}
>  		if (ret) {
>  			if (ret == -EBUSY)
>  				continue;
> @@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror,
>  		break;
>  	}
>  
> -	ret = dmirror_do_fault(dmirror, range);
> +	/* update device page table after migration */
> +	if (!migrate)
> +		ret = dmirror_do_fault(dmirror, range);
>  
>  	mutex_unlock(&dmirror->mutex);
>  out:

[Severity: High]
Can retrying inside dmirror_range_fault() when migrate is true cause a
self-deadlock and leak folio locks and references?

During the first call to hmm_range_fault(),
hmm_vma_handle_migrate_prepare() and hmm_vma_handle_migrate_prepare_pmd()
collect present pages by locking the folio, taking a folio reference,
installing a migration PTE, and setting HMM_PFN_VALID | HMM_PFN_MIGRATE in
range->hmm_pfns.

If hmm_range_fault() returns -EBUSY, or if hmm_range_fault() returns 0 and
mmu_interval_read_retry() trips in dmirror_range_fault() (for example,
after migrate_vma_split_folio() splits a PTE-mapped large folio and fires
MMU_NOTIFY_CLEAR, or due to a concurrent invalidation),
dmirror_range_fault() loops back via continue without returning to
do_fault_and_migrate() to abort and finalize the already-collected
migration entries.

On the next iteration of the loop in dmirror_range_fault(), calling
hmm_range_fault() -> hmm_range_fault_locked() zeroes range->hmm_pfns:

mm/hmm.c:hmm_range_fault_locked()
	if (hmm_select_migrate(range)) {
		WARN_ON_ONCE(range->pfn_flags_mask);
		WARN_ON_ONCE(range->dev_private_owner !=
			     range->migrate->pgmap_owner);
		memset(range->hmm_pfns, 0,
		       ((range->end - range->start) >> PAGE_SHIFT) *
		       sizeof(*range->hmm_pfns));
	}

When the retry walk visits the migration PTEs installed by the previous
attempt, hmm_pfn_collected(*hmm_pfn) evaluates to false in
hmm_vma_handle_pte() because *hmm_pfn was zeroed:

mm/hmm.c:hmm_vma_handle_pte()
		if (softleaf_is_migration(entry) && hmm_select_migrate(range) &&
		    hmm_pfn_collected(*hmm_pfn) &&
		    hmm_pfn_to_page(*hmm_pfn) == softleaf_to_page(entry))
			return 0;

It then treats its own migration entry as an external migration entry and
calls hmm_do_fault() -> handle_mm_fault() -> migration_entry_wait(),
waiting forever on the folio lock already held by current. In addition, if
any collected PTEs were unmapped before the retry walk (or if the retry
walk fails or times out before revisiting them), zeroing range->hmm_pfns
prevents hmm_vma_handle_migrate_prepare_rollback() and
migrate_hmm_range_setup() / migrate_vma_finalize() from unlocking those
folios and dropping their references.

Furthermore, if dmirror_range_fault() times out at time_after(jiffies,
timeout) before calling hmm_range_fault() on a subsequent subrange chunk,
do_fault_and_migrate() still runs migrate_hmm_range_setup(),
migrate_vma_pages(), and migrate_vma_finalize() on the stale src_pfns and
migrate->start/end state from the previous chunk.

Should dmirror_range_fault() return -EBUSY to do_fault_and_migrate() when
migrate is true so the caller can run the migrate_hmm_range_setup(),
migrate_vma_pages(), and migrate_vma_finalize() cleanup before retrying?

[ ... ]

> @@ -1377,6 +1398,127 @@ static int dmirror_migrate_to_device(struct dmirror *dmirror,
>  	return ret;
>  }
>  
> +static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range *range)
> +{
> +	struct migrate_vma *migrate = range->migrate;
> +	unsigned long timeout =
> +		jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
> +	int ret;
> +
> +	do {
> +		mmap_read_lock(dmirror->notifier.mm);
> +
> +		/* Fault-in pages for migration */
> +		ret = dmirror_range_fault(dmirror, range);
> +		/*
> +		 * Set this up even on error: hmm_range_fault() may have collected
> +		 * part of the range before failing.
> +		 */
> +		migrate_hmm_range_setup(range);
> +		if (ret) {
> +			/*
> +			 * dst[] entries are empty, so this marks every collected
> +			 * migration as failed. finalize then restores the source
> +			 * mappings and drops the associated locks/references.
> +			 */
> +			migrate_vma_pages(migrate);
> +			migrate_vma_finalize(migrate);
> +			goto unlock;
> +		}
> +
> +		pr_debug("Migrating from sys mem to device mem\n");
> +
> +		dmirror_migrate_alloc_and_copy(migrate, dmirror);
> +		migrate_vma_pages(migrate);
> +
> +		mutex_lock(&dmirror->mutex);
> +		if (mmu_interval_read_retry(&dmirror->notifier,
> +					    range->notifier_seq))
> +			ret = -EBUSY;
> +		else
> +			ret = __dmirror_migrate_map_locked(migrate, dmirror);
> +		mutex_unlock(&dmirror->mutex);
> +
> +		migrate_vma_finalize(migrate);
> +unlock:
> +		mmap_read_unlock(dmirror->notifier.mm);
> +	} while (ret == -EBUSY && !time_after(jiffies, timeout));
> +
> +	return ret;
> +}

[Severity: Medium]
As reported by Sashiko on v16 [1], when mmu_interval_read_retry() returns
true in do_fault_and_migrate() after migrate_vma_pages() has executed, can
this leave migrated device-private pages unmapped in dmirror->pt and cause
dmirror_fault_and_migrate_to_device() to fail with -ENOENT?

When mmu_interval_read_retry() trips here (for example, due to an
invalidation anywhere in the process mm since dmirror->notifier covers the
entire address space, or a partial zap, or split_huge_pmd() in
migrate_vma_insert_page()), __dmirror_migrate_map_locked() is skipped,
while migrate_vma_finalize() still commits any non-zapped migration PTEs to
device-private memory (and migrate_vma_insert_page() has already installed
device-private PTEs for pte_none or zero-page entries).

On the subsequent -EBUSY retry iteration of do_fault_and_migrate(), those
already-committed device-private pages are reported by hmm_vma_handle_pte()
as HMM_PFN_VALID without HMM_PFN_MIGRATE because migrate.flags only sets
MIGRATE_VMA_SELECT_SYSTEM, and dmirror_range_fault() skips
dmirror_do_fault() because HMM_PFN_REQ_MIGRATE is set.

Although the commit message notes that a real device would map committed
pages on its next fault, dmirror_fault_and_migrate_to_device() immediately
verifies the range by calling dmirror_do_read() directly without a fault
retry loop:

	mutex_lock(&dmirror->mutex);
	ret = dmirror_do_read(dmirror, start, end, &bounce);
	mutex_unlock(&dmirror->mutex);

Wouldn't dmirror_do_read() find no entry in dmirror->pt for those
already-migrated pages and fail the ioctl with -ENOENT?

> +
> +static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror,
> +					       struct hmm_dmirror_cmd *cmd)
> +{
> +	unsigned long start, size, end, next;
> +	unsigned long src_pfns[PFNS_ARRAY_SIZE] = { 0 };
> +	unsigned long dst_pfns[PFNS_ARRAY_SIZE] = { 0 };
> +	struct migrate_vma migrate = { 0 };
> +	struct hmm_range range = { 0 };
> +	struct dmirror_bounce bounce;

[Severity: Low]
Does allocating src_pfns[64] and dst_pfns[64] on the stack in
dmirror_fault_and_migrate_to_device() risk triggering -Wframe-larger-than
warnings?

On 64-bit architectures, src_pfns and dst_pfns alone consume 1024 bytes of
stack space alongside struct migrate_vma, struct hmm_range, and struct
dmirror_bounce (exceeding 1230 bytes of local variables before saved
registers and inlining do_fault_and_migrate()).

Commit 6046a3bed1c2 ("lib/test_hmm: reduce stack usage") previously had to
reduce the 64-element stack arrays in dmirror_migrate_to_device() after
hitting a 1408-byte stack frame build failure, and commit 775465fd26a3
("lib/test_hmm: use kvcalloc() for src/dst pfns allocation") moved
dmirror_migrate_to_device() and dmirror_migrate_to_system() to dynamic
allocation with kvcalloc(). Should dmirror_fault_and_migrate_to_device()
also use kvcalloc() or a smaller PFNS_ARRAY_SIZE?

[1] https://lore.kernel.org/r/20261021060506.2377545-11-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=10

Re: [PATCH v17 11/12] Documentation/mm/hmm: document migration through hmm_range_fault()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] mm/hmm: incomplete error rollback steps for migrate_hmm_range_setup() in docs
--

commit 908a5d19d6ca6398929f82b830d122f054aa64a3
Author: Mika Penttilä <mpenttil@redhat.com>

Documentation/mm/hmm: document migration through hmm_range_fault()

This commit documents the MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE flags for
migrate_vma_setup(), as well as driving the migration collection phase
directly from hmm_range_fault() via HMM_PFN_REQ_MIGRATE and
migrate_hmm_range_setup().

> diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
> index fc1b8dc198250..914ef6bfe8486 100644
> --- a/Documentation/mm/hmm.rst
> +++ b/Documentation/mm/hmm.rst

[ ... ]

> @@ -427,6 +434,44 @@ between device driver specific code and shared common code:
>  
>     The lock can now be released.
>  
> +Migration collection through hmm_range_fault()
> +==============================================

[ ... ]

> +After hmm_range_fault() returns, the driver calls::
> +
> +    void migrate_hmm_range_setup(struct hmm_range *range);
> +
> +to translate ``range->hmm_pfns`` into ``migrate->src[]`` (``migrate->dst[]`` is
> +zeroed) and initialize ``migrate->cpages`` and ``migrate->npages``. From here on
> +the ``range->migrate`` struct is ready for the rest of the flow (steps 3
> +onwards), i.e. migrate_vma_pages() and migrate_vma_finalize(). This should be
> +called even on error, since hmm_range_fault() may have collected part of the
> +range before failing.

[Severity: Low]
Could this wording be clarified regarding what needs to be called when
hmm_range_fault() returns an error?

In mm/migrate_device.c, migrate_hmm_range_setup() only translates
range->hmm_pfns into migrate->src[], zeroes migrate->dst[], initializes
migrate->cpages and migrate->npages, and calls migrate_vma_unmap(). Calling
migrate_hmm_range_setup() alone on error would leave migration PTEs installed
and leave collected folios isolated from the LRU, locked, and with elevated
reference counts.

On the other hand, if "This" is read as referring to the rest of the flow
(steps 3 onwards), step 3 in Documentation/mm/hmm.rst instructs the driver to
allocate destination pages, copy source pages, and populate migrate->dst[],
which would commit the partial migration rather than aborting it.

As described in the kernel-doc for hmm_range_fault() in mm/hmm.c:

 * On any other error, or when abandoning a retry, the caller must call
 * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination
 * entries, and migrate_vma_finalize() to abort the partial migration.

Should this paragraph explicitly state that on error the driver must call
migrate_hmm_range_setup(), skip step 3 so migrate->dst[] remains zeroed, and
still call migrate_vma_pages() and migrate_vma_finalize() to restore the
original PTEs and release the folio locks and references?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=11

Re: [PATCH v17 12/12] drm/nouveau/svm: add support for migrate on fault

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] drm/nouveau/dmem: unserialized access to shared channel drm->ttm.chan
- [Critical] drm/nouveau/dmem: folio state corrupted after nouveau_dmem_migrate_abort()
- [High] drm/nouveau/dmem: wrong folio pinned by nouveau_dmem_migrate_abort()
- [High] drm/nouveau: unreplayed GPU faults stall CE in nouveau_dmem_fence_poll()
- [High] drm/nouveau/svm: infinite GPU write fault loop in nouveau_dmem_migrate_fault()
- [Medium] drm/nouveau/svm: nouveau_svmm_bind() skips migration on migrate-on-fault
- [Low] drm/nouveau/svm: missing kernel-parameters.txt entry for svm_migrate_on_fault
- [Low] drm/nouveau: unused nouveau_dmem_fence_done() return and incomplete changelog
--

commit e30e6f260301c1f7393cda9890d6f9d63fe3f96e
Author: Mika Penttilä <mpenttil@redhat.com>

drm/nouveau/svm: add support for migrate on fault

This commit adds migrate-on-fault support to the nouveau SVM driver using
HMM_PFN_REQ_MIGRATE and a new nouveau_dmem_migrate_fault() helper, gated by
the svm_migrate_on_fault module parameter. It also introduces a migration
channel mutex and polling fence wait with abort handling for DMEM copies.

> diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c
> index e74d7bb975a86..0a5d04068321b 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c
[ ... ]
> @@ -76,6 +79,14 @@ struct nouveau_dmem_migrate {
>  	nouveau_migrate_copy_t copy_func;
>  	nouveau_clear_page_t clear_func;
>  	struct nouveau_channel *chan;
> +	/*
> +	 * Serializes the dmem users of chan (CPU faults in migrate_to_ram(),
> +	 * migrate on fault, bulk migration, eviction): the copy and clear
> +	 * commands they write to its push buffer and the fences they emit.
> +	 * Held only around those writes, never across page allocation.
> +	 * TTM serializes its own use of the channel with cli->mutex.
> +	 */
> +	struct mutex lock;
>  };
[ ... ]
> +static int nouveau_dmem_copy(struct nouveau_drm *drm, u64 npages,
> +			     enum nouveau_aper dst_aper, u64 dst_addr,
> +			     enum nouveau_aper src_aper, u64 src_addr)
> +{
> +	int ret;
> +
> +	mutex_lock(&drm->dmem->migrate.lock);
> +	ret = drm->dmem->migrate.copy_func(drm, npages, dst_aper, dst_addr,
> +					   src_aper, src_addr);
> +	mutex_unlock(&drm->dmem->migrate.lock);
> +	return ret;
> +}
> +
> +static int nouveau_dmem_clear(struct nouveau_drm *drm, u32 length,
> +			      enum nouveau_aper dst_aper, u64 dst_addr)
> +{
> +	int ret;
> +
> +	mutex_lock(&drm->dmem->migrate.lock);
> +	ret = drm->dmem->migrate.clear_func(drm, length, dst_aper, dst_addr);
> +	mutex_unlock(&drm->dmem->migrate.lock);
> +	return ret;
> +}
> +
> +static int nouveau_dmem_fence_new(struct nouveau_drm *drm,
> +				  struct nouveau_fence **fence)
> +{
> +	int ret;
> +
> +	mutex_lock(&drm->dmem->migrate.lock);
> +	ret = nouveau_fence_new(fence, drm->dmem->migrate.chan);
> +	mutex_unlock(&drm->dmem->migrate.lock);
> +	return ret;
> +}

[Severity: Critical]
As reported by Sashiko on v16 [1], in drivers/gpu/drm/nouveau/nouveau_dmem.c,
nouveau_dmem_migrate_init() sets drm->dmem->migrate.chan = drm->ttm.chan, so
DMEM and TTM share the same struct nouveau_channel instance:

    drm->dmem->migrate.copy_func = nvc0b5_migrate_copy;
    drm->dmem->migrate.clear_func = nvc0b5_migrate_clear;
    drm->dmem->migrate.chan = drm->ttm.chan;

Meanwhile, in drivers/gpu/drm/nouveau/nouveau_bo.c, nouveau_bo_move_m2mf()
uses drm->ttm.chan under cli->mutex without taking drm->dmem->migrate.lock:

    struct nouveau_channel *chan = drm->ttm.chan;
    struct nouveau_cli *cli = chan->cli;
    ...
    if (drm_drv_uses_atomic_modeset(drm->dev))
        mutex_lock(&cli->mutex);
    else
        mutex_lock_nested(&cli->mutex, SINGLE_DEPTH_NESTING);

Because nouveau_dmem_copy(), nouveau_dmem_clear(), and
nouveau_dmem_fence_new() take only drm->dmem->migrate.lock while
nouveau_bo_move_m2mf() takes only cli->mutex on the same underlying channel,
can concurrent DMEM migrations (from nouveau_dmem_migrate_fault(),
nouveau_dmem_migrate_to_ram(), nouveau_dmem_migrate_vma(), or
nouveau_dmem_evict_chunk()) and TTM buffer moves race on chan->chan.push,
interleave NVA0B5 copy engine methods in the pushbuffer, and corrupt
++fctx->sequence and fctx->pending in nouveau_fence_emit()?

> @@ -139,10 +185,12 @@ static void nouveau_dmem_folio_free(struct folio *folio)
>  	spin_unlock(&dmem->lock);
>  }
>  
> -static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
> +static int nouveau_dmem_fence_done(struct nouveau_fence **fence)
>  {
> +	int ret = 0;
> +
>  	if (fence) {
> -		nouveau_fence_wait(*fence, true, false);
> +		ret = nouveau_fence_wait(*fence, true, false);
>  		nouveau_fence_unref(fence);
>  	} else {
>  		/*
> @@ -150,6 +198,54 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
>  		 * the hmem object.
>  		 */
>  	}
> +
> +	/* 0 on completion, -EBUSY on timeout, or other errno; propagated. */
> +	return ret;
>  }

[Severity: Low]
This isn't a bug, but nouveau_dmem_fence_done() is changed to return int
while its remaining callers in drivers/gpu/drm/nouveau/nouveau_dmem.c
(nouveau_dmem_evict_chunk() and nouveau_dmem_migrate_chunk()) ignore the
return value.

Also, the commit message only describes adding HMM_PFN_REQ_MIGRATE support,
nouveau_dmem_migrate_fault(), and the svm_migrate_on_fault parameter, without
mentioning the addition of drm->dmem->migrate.lock or the rework of
nouveau_dmem_migrate_to_ram() to use nouveau_dmem_fence_poll() and
nouveau_dmem_migrate_abort() even when svm_migrate_on_fault is off. Could the
unused return value be dropped and the changelog updated to cover these
changes?

> +/*
> + * Poll the copy fence instead of the lazy (interrupt-driven) wait since
> + * completion IRQ is randomly missed, hitting the timeout.
> + *
> + * Returns 0 on completion, -EBUSY on timeout. Consumes the fence reference.
> + */
> +static int nouveau_dmem_fence_poll(struct nouveau_fence **fence)
> +{
> +	unsigned long deadline = jiffies + 30 * HZ;
> +	int ret = 0;
> +
> +	while (!nouveau_fence_done(*fence)) {
> +		if (time_after(jiffies, deadline)) {
> +			ret = -EBUSY;
> +			break;
> +		}
> +		schedule_timeout_uninterruptible(msecs_to_jiffies(1));
> +	}
> +	nouveau_fence_unref(fence);
> +
> +	return ret;
> +}

[Severity: High]
As noted in nouveau_dmem_migrate_fault(), when a replayable GPU fault is
pending, a copy engine transfer does not make progress until
nouveau_svm_fault_replay() is issued. Can nouveau_dmem_fence_poll() stall
for 30 seconds and time out in two scenarios?

First, in nouveau_dmem_migrate_to_ram(), migrate_vma_setup() locks sfolio
and installs a migration PTE, nouveau_svmm_invalidate() unmaps the page from
the GPU, and nouveau_dmem_copy_folio() starts the copy before
nouveau_dmem_fence_poll() waits for completion. If the GPU faults on that
invalidated address while the copy is in flight, the pending GPU fault stalls
the copy engine, while the fault worker in nouveau_svm_fault() blocks in
hmm_range_fault() -> migration_entry_wait() waiting for sfolio to be unlocked
by migrate_vma_finalize(). Because nouveau_dmem_migrate_to_ram() never calls
nouveau_svm_fault_replay(), won't nouveau_dmem_fence_poll() stall until the
30-second timeout, leak the page in nouveau_dmem_migrate_abort(), and return
VM_FAULT_SIGBUS?

Second, in nouveau_dmem_migrate_fault(), nouveau_svm_fault_replay() is called
once before nouveau_dmem_fence_poll(), while the GPU page table entry is
still unmapped (__nouveau_pfns_map_locked() runs only after the fence completes).
If the replayed GPU access or another warp immediately refaults on the still
unmapped address before the copy finishes, won't the GPU re-enter the faulted
state and stall the copy engine while the fault worker is blocked inside
nouveau_dmem_fence_poll()?

> +/*
> + * The copy could not be confirmed (fence timeout or no fence): don't migrate,
> + * so finalize restores the original ptes and the source stays valid. The copy
> + * engine may still be using the pages, so keep the destination folios (an
> + * extra reference that finalize won't drop) and let the caller skip the DMA
> + * unmap: leaking is safer than freeing memory a wedged engine might write.
> + */
> +static void nouveau_dmem_migrate_abort(struct nouveau_drm *drm,
> +				       struct migrate_vma *args)
> +{
> +	unsigned long i;
> +
> +	for (i = 0; i < args->npages; i++) {
> +		struct page *dpage = migrate_pfn_to_page(args->dst[i]);
> +
> +		args->src[i] &= ~MIGRATE_PFN_MIGRATE;
> +		if (dpage)
> +			folio_get(page_folio(dpage));
> +	}
> +	NV_ERROR(drm, "svm: copy not confirmed, migration aborted, pages leaked\n");
> +}
[ ... ]
> @@ -264,11 +361,21 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
>  		goto done;
>  	}
>  
> -	nouveau_fence_new(&fence, dmem->migrate.chan);
> +	/*
> +	 * Commit the migration only once the copy is confirmed. Without a
> +	 * fence the copy can't be confirmed: treat it as failed.
> +	 */
> +	fret = nouveau_dmem_fence_new(drm, &fence);
> +	if (!fret)
> +		fret = nouveau_dmem_fence_poll(&fence);
> +	if (fret) {
> +		nouveau_dmem_migrate_abort(drm, &args);
> +		ret = VM_FAULT_SIGBUS;
> +	} else {
> +		dma_unmap_page(drm->dev->dev, dma_info.dma_addr, dma_info.size,
> +			       DMA_BIDIRECTIONAL);
> +	}
>  	migrate_vma_pages(&args);
> -	nouveau_dmem_fence_done(&fence);
> -	dma_unmap_page(drm->dev->dev, dma_info.dma_addr, dma_info.size,
> -				DMA_BIDIRECTIONAL);
>  done:
>  	migrate_vma_finalize(&args);

[Severity: Critical]
As reported by Sashiko on v16 [2], when nouveau_dmem_fence_new() or
nouveau_dmem_fence_poll() fails in nouveau_dmem_migrate_to_ram() or
nouveau_dmem_migrate_fault(), nouveau_dmem_migrate_abort() clears
MIGRATE_PFN_MIGRATE in args->src[i] while leaving args->dst[i] non-zero, and
then the caller invokes migrate_vma_pages() and migrate_vma_finalize().

In mm/migrate_device.c:__migrate_device_pages(), MIGRATE_PFN_MIGRATE is only
checked in the !page branch; when both page (from src_pfns[i]) and newpage
(from dst_pfns[i]) are non-NULL, __migrate_device_pages() still executes:

    folio = page_folio(migrate_pfn_to_page(src_pfns[i+j]));
    newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j]));
    mapping = folio_mapping(folio);
    r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt);
    if (r)
        src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE;
    else
        folio_migrate_flags(newfolio, folio);

This removes the source folio from the swap cache via folio_free_swap(),
migrates mapping and page flags to newfolio, and calls
mem_cgroup_migrate(folio, newfolio), which zeroes folio->memcg_data on the
source folio. Then migrate_vma_finalize() sees !(src_pfns[i] &
MIGRATE_PFN_MIGRATE) and restores the CPU PTEs to the corrupted source folio
whose memcg charge and swapcache state were already moved to the leaked
destination folio.

Should nouveau_dmem_migrate_abort() instead unlock and release/leak dst[i]
and clear args->dst[i] = 0 (or skip migrate_vma_pages() altogether when
aborting) so __migrate_device_pages() does not touch the source folio?

[Severity: High]
In addition, when nouveau_dmem_migrate_abort() is called from
nouveau_dmem_migrate_fault() during system-to-VRAM migration, does it pin the
wrong folio?

In nouveau_dmem_migrate_fault(), migrate.src[i] is the host system RAM folio
that was DMA-mapped into dma_info by nouveau_dmem_migrate_copy_one(), whereas
migrate.dst[i] is a device-private VRAM folio. When fret != 0,
nouveau_dmem_migrate_fault() skips dma_unmap_page() on dma_info because the
copy engine may still be accessing the host DMA mapping, but
nouveau_dmem_migrate_abort() only calls folio_get() on migrate.dst[i].

Won't migrate_vma_finalize() drop the migration reference on the DMA-mapped
system RAM folio in migrate.src[i] (allowing it to be freed back to the page
allocator while still DMA-mapped), while the extra folio_get() on the
unmapped device-private folio in migrate.dst[i] prevents chunk->callocated
from ever reaching 0 and causes memunmap_pages(&chunk->pagemap) to hang on
driver teardown in nouveau_dmem_evict_chunk()?

[ ... ]
> @@ -832,6 +956,238 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
[ ... ]
> +		mutex_lock(&svmm->mutex);
> +		if (fret) {
> +			/* Copy did not complete; don't map a half-copied page. */
> +			ret = -EIO;
> +		} else if (mmu_interval_read_retry(notifier, range.notifier_seq)) {
> +			ret = -EBUSY;
> +		} else {
> +			unsigned long map_addr = migrate.start;
> +			unsigned int page_shift = PAGE_SHIFT + order;
> +
> +			/*
> +			 * If nothing migrated might still be able to mirror.
> +			 * In that case use the snapshot hmm_pfn.
> +			 * npages == 1 always today.
> +			 */
> +			if (npages == 1 && !(pfns[0] & NVIF_VMM_PFNMAP_V0_V))
> +				pfns[0] = nouveau_dmem_hmm_pfn_to_map(hmm_pfn0,
> +						start, &map_addr, &page_shift);
> +
> +			/*
> +			 * Nothing to map: leave the GPU mapping alone, the
> +			 * caller falls back to nouveau_range_fault().
> +			 */
> +			if (pfns[0] & NVIF_VMM_PFNMAP_V0_V) {
> +				/* On failure the caller falls back, then cancels */
> +				ret = __nouveau_pfns_map_locked(svmm, map_addr,
> +								pfns, n,
> +								page_shift);
> +				if (!ret) {
> +					*out_pfn0 = pfns[0];
> +					*out_addr = map_addr;
> +					*out_shift = page_shift;
> +				}
> +			}
> +		}
> +		mutex_unlock(&svmm->mutex);

[Severity: High]
What happens in nouveau_dmem_migrate_fault() when a GPU write fault
(hmm_flags containing HMM_PFN_REQ_WRITE) hits a read-only device-private
VRAM PTE owned by drm->dev (for example, a page migrated to VRAM on an
earlier GPU read fault when the CPU PTE was read-only, or after fork() or
mprotect(PROT_READ))?

In mm/hmm.c:hmm_vma_handle_pte(), when a non-present PTE is a device-private
entry owned by range->dev_private_owner, hmm_range_fault() returns 0 with
HMM_PFN_VALID but without HMM_PFN_WRITE (because
softleaf_is_device_private_write(entry) is false) and without
HMM_PFN_MIGRATE.

Then in nouveau_dmem_migrate_fault(), nothing migrates, so it falls back to
nouveau_dmem_hmm_pfn_to_map(hmm_pfn0, ...), which encodes pfns[0] with
NVIF_VMM_PFNMAP_V0_V | NVIF_VMM_PFNMAP_V0_VRAM but without
NVIF_VMM_PFNMAP_V0_W. __nouveau_pfns_map_locked() installs a read-only GPU
PTE and returns 0, and nouveau_svm_fault() replays the fault without falling
back to nouveau_range_fault(). Won't the GPU write instruction fault again
indefinitely in an infinite fault-replay loop because write permission was
requested in hmm_flags but not granted in hmm_pfn0?

> diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
> index c519c28164fa0..6e109274bd77a 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_svm.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
[ ... ]
> @@ -78,6 +79,13 @@ struct nouveau_svm {
>  #define SVM_DBG(s,f,a...) NV_DEBUG((s)->drm, "svm: "f"\n", ##a)
>  #define SVM_ERR(s,f,a...) NV_WARN((s)->drm, "svm: "f"\n", ##a)
>  
> +/* When set, migrate the faulting range to VRAM on GPU fault (default off). */
> +static bool nouveau_svm_migrate_on_fault;
> +module_param_named(svm_migrate_on_fault, nouveau_svm_migrate_on_fault, bool,
> +		   0644);
> +MODULE_PARM_DESC(svm_migrate_on_fault,
> +		 "migrate system memory to VRAM on GPU fault (default off)");

[Severity: Low]
As reported by Sashiko on v16 [3], this adds the nouveau.svm_migrate_on_fault
module parameter via module_param_named() in drivers/gpu/drm/nouveau/nouveau_svm.c
without an entry in Documentation/admin-guide/kernel-parameters.txt (noting
that CONFIG_DRM_NOUVEAU_SVM is marked experimental and depends on STAGING in
drivers/gpu/drm/nouveau/Kconfig). Should this parameter be documented in
Documentation/admin-guide/kernel-parameters.txt?

> @@ -182,9 +190,10 @@ nouveau_svmm_bind(struct drm_device *dev, void *data,
>  
>  		addr = max(addr, vma->vm_start);
>  		next = min(vma->vm_end, end);
> -		/* This is a best effort so we ignore errors */
> -		nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma, addr,
> -					 next);
> +		/* Best effort; skipped when migrate-on-fault does the migration. */
> +		if (!nouveau_svm_migrate_on_fault)
> +			nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma,
> +						 addr, next);
>  		addr = next;
>  	}

[Severity: Medium]
In drivers/gpu/drm/nouveau/nouveau_svm.c, why does nouveau_svmm_bind() skip
nouveau_dmem_migrate_vma() when nouveau_svm_migrate_on_fault is enabled
(while still taking mmap_read_lock(mm) and walking the VMAs in the range)?

If pages in the bound range are already mapped to system memory in the GPU
page tables (for example from a prior FAULT_ACCESS_PREFETCH fault, an atomic
fault in nouveau_atomic_range_fault(), or an in-place fallback mapping in
nouveau_dmem_migrate_fault()), subsequent GPU accesses will not fault, so
skipping nouveau_dmem_migrate_vma() prevents explicit
DRM_IOCTL_NOUVEAU_SVM_BIND requests from ever migrating those pages to VRAM.
Furthermore, nouveau_dmem_migrate_vma() migrates compound PMD THPs with
MIGRATE_VMA_SELECT_COMPOUND, whereas deferring to nouveau_dmem_migrate_fault()
faults one 4KB page at a time without MIGRATE_VMA_SELECT_COMPOUND and splits
2MB THPs. Is it intentional that enabling svm_migrate_on_fault turns the
migration ioctl into a no-op?

[1] https://lore.kernel.org/r/20261021060506.2377545-13-mpenttil@redhat.com
[2] https://lore.kernel.org/r/20261021060506.2377545-13-mpenttil@redhat.com
[3] https://lore.kernel.org/r/20261021060506.2377545-13-mpenttil@redhat.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103061629.972669-1-mpenttil@redhat.com?part=12