[PATCH 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[Usama Arif <usama.arif@linux.dev>]

madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
madvise_free_huge_pmd() takes pmd_trans_huge_lock().
pmd_is_huge() returns true for a device-private PMD, so orig_pmd can
be device-private and hit the VM_BUG_ON() on the !pmd_present() branch.

Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
migrate_vma_pages() flips the PMD to a device-private entry between the
caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().

Skip device-private PMDs after taking the lock, before the
!pmd_present() check.

Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 mm/huge_memory.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c0892cc533a9..cfce9f31b30e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2296,6 +2296,9 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	if (is_huge_zero_pmd(orig_pmd))
 		goto out;
 
+	if (pmd_is_device_private_entry(orig_pmd))
+		goto out;
+
 	if (unlikely(!pmd_present(orig_pmd))) {
 		VM_BUG_ON(thp_migration_supported() &&
 				  !pmd_is_migration_entry(orig_pmd));
-- 
2.53.0-Meta