[PATCH v3 0/3] mm: handle device-private PMDs in walk callbacks

[PATCH v3 0/3] mm: handle device-private PMDs in walk callbacks

[Usama Arif]

Since commit 368076f52ebe ("mm/huge_memory: add device-private THP support
to PMD operations") a PMD may hold a device-private swap entry whenever
an HMM-based GPU driver migrates an anonymous THP folio to device memory
via migrate_vma_pages().

pmd_trans_huge_lock() succeeds for such PMDs (pmd_is_huge() returns true
for any non-present, non-none huge PMD), so several MM walk callbacks
that used to assume present THP or migration entry are now reachable with
a device-private PMD. The results range from a VM_BUG_ON() firing on debug
kernels, to an oops on a bogus vmemmap dereference, to silently isolating
an unrelated live folio from LRU in the aliasing case.

---
v2 -> v3: https://lore.kernel.org/all/20260708122040.861335-1-usama.arif@linux.dev/
- Patch 1: use pmdp_get() instead of *pmd. (Zi)
- Patches 2 and 3: drop the thp_migration_supported() guard and
  downgrade VM_BUG_ON to VM_WARN_ON_ONCE(). thp_migration_supported()
  expands to IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
  pmd_is_migration_entry() and pmd_is_device_private_entry() already
  return false without that config, so the guard suppresses only the
  case where the warning would already be silent. (David)

v1 -> v2: https://lore.kernel.org/all/20260707135255.292870-1-usama.arif@linux.dev/
- Patch 1 now gates queue_folios_pmd() on !pmd_present() instead of
  checking only pmd_is_device_private_entry(). This matches the PTE path,
  keeps migration entries counted in qp->nr_failed, and skips other
  non-present PMDs such as device-private entries before calling
  pmd_folio(). (Joshua)
- Patches 2 and 3 now fold device-private PMD handling into the existing
  !pmd_present() VM_BUG_ON() condition instead of using a separate early
  pmd_is_device_private_entry() check. (Joshua)
- cc stable (Zi)
 
Usama Arif (3):
  mm/mempolicy: skip non-present PMDs when queueing folios
  mm/madvise: skip device-private PMDs in cold and pageout walks
  mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

 mm/huge_memory.c | 4 ++--
 mm/madvise.c     | 4 ++--
 mm/mempolicy.c   | 8 +++++---
 3 files changed, 9 insertions(+), 7 deletions(-)

-- 
2.53.0-Meta

[PATCH v3 1/3] mm/mempolicy: skip non-present PMDs when queueing folios

[Usama Arif]

queue_folios_pmd() is called under pmd_trans_huge_lock(), whose
pmd_is_huge() check returns true for any non-present, non-none PMD
softleaf. Passing such a PMD to pmd_folio() treats the softleaf encoding
as a hardware PFN and can return a bogus folio pointer.

Mirror queue_folios_pte_range(): handle non-present entries before
looking up a folio. Keep migration entries counted as failures, but skip
other non-present PMDs such as device-private entries.

Potential trigger: an HMM-based GPU driver migrates an anonymous THP
folio to device memory via migrate_vma_pages(), leaving a device-private
PMD. Userspace then calls mbind(), migrate_pages() or
set_mempolicy_home_node() on that range.

Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
Cc: <stable@vger.kernel.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 mm/mempolicy.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 914f81863db5..69a00a324ef5 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -654,12 +654,14 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
 {
 	struct folio *folio;
 	struct queue_pages *qp = walk->private;
+	pmd_t pmdval = pmdp_get(pmd);
 
-	if (unlikely(pmd_is_migration_entry(*pmd))) {
-		qp->nr_failed++;
+	if (unlikely(!pmd_present(pmdval))) {
+		if (pmd_is_migration_entry(pmdval))
+			qp->nr_failed++;
 		return;
 	}
-	folio = pmd_folio(*pmd);
+	folio = pmd_folio(pmdval);
 	if (is_huge_zero_folio(folio)) {
 		walk->action = ACTION_CONTINUE;
 		return;
-- 
2.53.0-Meta

[PATCH v3 2/3] mm/madvise: skip device-private PMDs in cold and pageout walks

[Usama Arif]

madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
pmd_is_huge() check returns true for a device-private PMD. The subsequent
!pmd_present() branch has a VM_BUG_ON() asserting migration is the only
allowed non-present case; a device-private PMD trips it.

Skip device-private PMDs in that non-present branch and continue to
huge_unlock before calling pmd_folio(). Downgrade the check to
VM_WARN_ON_ONCE() so an unexpected PMD softleaf logs a warning rather
than panicking. Drop the thp_migration_supported() guard: it expands to
IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
pmd_is_migration_entry() and pmd_is_device_private_entry() already
return false when that config is not selected, so the guard suppresses
only the case where the warning would already be silent.

Potential trigger: an HMM-based GPU driver races with
madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
migrate_vma_pages() flips the PMD to a device-private entry before the
PMD lock is acquired.

Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
Cc: <stable@vger.kernel.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 mm/madvise.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 9292f60b19aa..c557023c3fad 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -388,8 +388,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			goto huge_unlock;
 
 		if (unlikely(!pmd_present(orig_pmd))) {
-			VM_BUG_ON(thp_migration_supported() &&
-					!pmd_is_migration_entry(orig_pmd));
+			VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
+					!pmd_is_device_private_entry(orig_pmd));
 			goto huge_unlock;
 		}
 
-- 
2.53.0-Meta

[PATCH v3 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[Usama Arif]

madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
madvise_free_huge_pmd() takes pmd_trans_huge_lock(). pmd_is_huge()
returns true for a device-private PMD, so orig_pmd can be device-private
and enter the !pmd_present() branch.

Skip device-private PMDs in that non-present branch and continue to out
before calling pmd_folio(). Downgrade the check to VM_WARN_ON_ONCE() so
an unexpected PMD softleaf logs a warning rather than panicking. Drop
the thp_migration_supported() guard: it expands to
IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
pmd_is_migration_entry() and pmd_is_device_private_entry() already
return false when that config is not selected, so the guard suppresses
only the case where the warning would already be silent.

Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
migrate_vma_pages() flips the PMD to a device-private entry between the
caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().

Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
Cc: <stable@vger.kernel.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 mm/huge_memory.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c0892cc533a9..7ae21b006b68 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2297,8 +2297,8 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		goto out;
 
 	if (unlikely(!pmd_present(orig_pmd))) {
-		VM_BUG_ON(thp_migration_supported() &&
-				  !pmd_is_migration_entry(orig_pmd));
+		VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
+				!pmd_is_device_private_entry(orig_pmd));
 		goto out;
 	}
 
-- 
2.53.0-Meta

Re: [PATCH v3 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[David Hildenbrand (Arm)]

On 7/10/26 12:55, Usama Arif wrote:
> madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
> madvise_free_huge_pmd() takes pmd_trans_huge_lock(). pmd_is_huge()
> returns true for a device-private PMD, so orig_pmd can be device-private
> and enter the !pmd_present() branch.
> 
> Skip device-private PMDs in that non-present branch and continue to out
> before calling pmd_folio(). Downgrade the check to VM_WARN_ON_ONCE() so
> an unexpected PMD softleaf logs a warning rather than panicking. Drop
> the thp_migration_supported() guard: it expands to
> IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
> pmd_is_migration_entry() and pmd_is_device_private_entry() already
> return false when that config is not selected, so the guard suppresses
> only the case where the warning would already be silent.
> 
> Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
> migrate_vma_pages() flips the PMD to a device-private entry between the
> caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().
> 
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Balbir Singh <balbirs@nvidia.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v3 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[David Hildenbrand (Arm)]

On 7/10/26 12:55, Usama Arif wrote:
> madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
> madvise_free_huge_pmd() takes pmd_trans_huge_lock(). pmd_is_huge()
> returns true for a device-private PMD, so orig_pmd can be device-private
> and enter the !pmd_present() branch.
> 
> Skip device-private PMDs in that non-present branch and continue to out
> before calling pmd_folio(). Downgrade the check to VM_WARN_ON_ONCE() so
> an unexpected PMD softleaf logs a warning rather than panicking. Drop
> the thp_migration_supported() guard: it expands to
> IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
> pmd_is_migration_entry() and pmd_is_device_private_entry() already
> return false when that config is not selected, so the guard suppresses
> only the case where the warning would already be silent.
> 
> Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
> migrate_vma_pages() flips the PMD to a device-private entry between the
> caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().
> 
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Balbir Singh <balbirs@nvidia.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v3 0/3] mm: handle device-private PMDs in walk callbacks

[Andrew Morton]

On Fri, 10 Jul 2026 03:55:20 -0700 Usama Arif <usama.arif@linux.dev> wrote:

> Since commit 368076f52ebe ("mm/huge_memory: add device-private THP support
> to PMD operations") a PMD may hold a device-private swap entry whenever
> an HMM-based GPU driver migrates an anonymous THP folio to device memory
> via migrate_vma_pages().
> 
> pmd_trans_huge_lock() succeeds for such PMDs (pmd_is_huge() returns true
> for any non-present, non-none huge PMD), so several MM walk callbacks
> that used to assume present THP or migration entry are now reachable with
> a device-private PMD. The results range from a VM_BUG_ON() firing on debug
> kernels, to an oops on a bogus vmemmap dereference, to silently isolating
> an unrelated live folio from LRU in the aliasing case.

Added, thanks.

I didn't add these as hotfixes - the Fixes: commit is somewhat old. 
Feel free to disagree with this!

Sashiko went nuts over possible pre-existing issues:

	https://sashiko.dev/#/patchset/20260710105557.1987433-1-usama.arif@linux.dev

Re: [PATCH v3 1/3] mm/mempolicy: skip non-present PMDs when queueing folios

[Gregory Price]

On Fri, Jul 10, 2026 at 03:55:21AM -0700, Usama Arif wrote:
> queue_folios_pmd() is called under pmd_trans_huge_lock(), whose
> pmd_is_huge() check returns true for any non-present, non-none PMD
> softleaf. Passing such a PMD to pmd_folio() treats the softleaf encoding
> as a hardware PFN and can return a bogus folio pointer.
> 
> Mirror queue_folios_pte_range(): handle non-present entries before
> looking up a folio. Keep migration entries counted as failures, but skip
> other non-present PMDs such as device-private entries.
> 
> Potential trigger: an HMM-based GPU driver migrates an anonymous THP
> folio to device memory via migrate_vma_pages(), leaving a device-private
> PMD. Userspace then calls mbind(), migrate_pages() or
> set_mempolicy_home_node() on that range.
> 
> Reported-by: sashiko-bot <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Reviewed-by: Gregory Price <gourry@gourry.net>

Re: [PATCH v3 2/3] mm/madvise: skip device-private PMDs in cold and pageout walks

[Gregory Price]

On Fri, Jul 10, 2026 at 03:55:22AM -0700, Usama Arif wrote:
> madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
> pmd_is_huge() check returns true for a device-private PMD. The subsequent
> !pmd_present() branch has a VM_BUG_ON() asserting migration is the only
> allowed non-present case; a device-private PMD trips it.
> 
> Skip device-private PMDs in that non-present branch and continue to
> huge_unlock before calling pmd_folio(). Downgrade the check to
> VM_WARN_ON_ONCE() so an unexpected PMD softleaf logs a warning rather
> than panicking. Drop the thp_migration_supported() guard: it expands to
> IS_ENABLED(CONFIG_ARCH_SUPPORTS_PMD_SOFTLEAF), and both
> pmd_is_migration_entry() and pmd_is_device_private_entry() already
> return false when that config is not selected, so the guard suppresses
> only the case where the warning would already be silent.
> 
> Potential trigger: an HMM-based GPU driver races with
> madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
> migrate_vma_pages() flips the PMD to a device-private entry before the
> PMD lock is acquired.
> 
> Reported-by: sashiko-bot <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Balbir Singh <balbirs@nvidia.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Reviewed-by: Gregory Price <gourry@gourry.net>

Re: [PATCH v3 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[Gregory Price]

On Fri, Jul 10, 2026 at 03:55:23AM -0700, Usama Arif wrote:
> 
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Balbir Singh <balbirs@nvidia.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Reviewed-by: Gregory Price <gourry@gourry.net>

> ---
>  mm/huge_memory.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index c0892cc533a9..7ae21b006b68 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2297,8 +2297,8 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  		goto out;
>  
>  	if (unlikely(!pmd_present(orig_pmd))) {
> -		VM_BUG_ON(thp_migration_supported() &&
> -				  !pmd_is_migration_entry(orig_pmd));
> +		VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
> +				!pmd_is_device_private_entry(orig_pmd));

I just realized, are there softleaf entries we wouldn't want to WARN on?
If not, should all three of these patches just be something like
pmd_is_softleaf()?

x_is_softleaf() does not exist, just curious if it should, and not
worth holding up the patch.

~Gregory

Re: [PATCH v3 0/3] mm: handle device-private PMDs in walk callbacks

[Balbir Singh]

On Fri, Jul 10, 2026 at 04:53:48PM -0700, Andrew Morton wrote:
> On Fri, 10 Jul 2026 03:55:20 -0700 Usama Arif <usama.arif@linux.dev> wrote:
> 
> > Since commit 368076f52ebe ("mm/huge_memory: add device-private THP support
> > to PMD operations") a PMD may hold a device-private swap entry whenever
> > an HMM-based GPU driver migrates an anonymous THP folio to device memory
> > via migrate_vma_pages().
> > 
> > pmd_trans_huge_lock() succeeds for such PMDs (pmd_is_huge() returns true
> > for any non-present, non-none huge PMD), so several MM walk callbacks
> > that used to assume present THP or migration entry are now reachable with
> > a device-private PMD. The results range from a VM_BUG_ON() firing on debug
> > kernels, to an oops on a bogus vmemmap dereference, to silently isolating
> > an unrelated live folio from LRU in the aliasing case.
> 
> Added, thanks.
>

Thanks Andrew!

> I didn't add these as hotfixes - the Fixes: commit is somewhat old. 
> Feel free to disagree with this!
> 
> Sashiko went nuts over possible pre-existing issues:
> 
> 	https://sashiko.dev/#/patchset/20260710105557.1987433-1-usama.arif@linux.dev
> 

The pointed out seems plausible? I am working on some tests to catch
more errors via hmm-tests.

Balbir

Re: [PATCH v3 1/3] mm/mempolicy: skip non-present PMDs when queueing folios

[Balbir Singh]

On Fri, Jul 10, 2026 at 03:55:21AM -0700, Usama Arif wrote:
> queue_folios_pmd() is called under pmd_trans_huge_lock(), whose
> pmd_is_huge() check returns true for any non-present, non-none PMD
> softleaf. Passing such a PMD to pmd_folio() treats the softleaf encoding
> as a hardware PFN and can return a bogus folio pointer.
> 
> Mirror queue_folios_pte_range(): handle non-present entries before
> looking up a folio. Keep migration entries counted as failures, but skip
> other non-present PMDs such as device-private entries.
> 
> Potential trigger: an HMM-based GPU driver migrates an anonymous THP
> folio to device memory via migrate_vma_pages(), leaving a device-private
> PMD. Userspace then calls mbind(), migrate_pages() or
> set_mempolicy_home_node() on that range.
> 
> Reported-by: sashiko-bot <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  mm/mempolicy.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 914f81863db5..69a00a324ef5 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -654,12 +654,14 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
>  {
>  	struct folio *folio;
>  	struct queue_pages *qp = walk->private;
> +	pmd_t pmdval = pmdp_get(pmd);
>  
> -	if (unlikely(pmd_is_migration_entry(*pmd))) {
> -		qp->nr_failed++;
> +	if (unlikely(!pmd_present(pmdval))) {
> +		if (pmd_is_migration_entry(pmdval))
> +			qp->nr_failed++;
>  		return;
>  	}
> -	folio = pmd_folio(*pmd);
> +	folio = pmd_folio(pmdval);
>  	if (is_huge_zero_folio(folio)) {
>  		walk->action = ACTION_CONTINUE;
>  		return;
> -- 

Reviewed-by: Balbir Singh <balbirs@nvidia.com>