Re: [PATCH 3/3] mm/huge_memory: skip device-private PMDs in madvise_free_huge_pmd

[Balbir Singh <balbirs@nvidia.com>]

On Tue, Jul 07, 2026 at 06:45:09AM -0700, Usama Arif wrote:
> madvise_free_pte_range() checks pmd_trans_huge(*pmd) unlocked, then
> madvise_free_huge_pmd() takes pmd_trans_huge_lock().
> pmd_is_huge() returns true for a device-private PMD, so orig_pmd can
> be device-private and hit the VM_BUG_ON() on the !pmd_present() branch.
> 
> Potential trigger: an HMM-based GPU driver races with madvise(MADV_FREE):
> migrate_vma_pages() flips the PMD to a device-private entry between the
> caller's pmd_trans_huge() check and the callee's pmd_trans_huge_lock().
> 
> Skip device-private PMDs after taking the lock, before the
> !pmd_present() check.
> 
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  mm/huge_memory.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index c0892cc533a9..cfce9f31b30e 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2296,6 +2296,9 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	if (is_huge_zero_pmd(orig_pmd))
>  		goto out;
>  
> +	if (pmd_is_device_private_entry(orig_pmd))
> +		goto out;
> +
>  	if (unlikely(!pmd_present(orig_pmd))) {
>  		VM_BUG_ON(thp_migration_supported() &&
>  				  !pmd_is_migration_entry(orig_pmd));
>

The change makes sense, I spent some time thinking if we should make
madvise pmd free work for THP PMD entries, but that should be a separate
patchset, it is much more complex and involved.

Reviewed-by: Balbir Singh <balbirs@nvidia.com>