Re: [PATCH 2/3] mm/madvise: skip device-private PMDs in cold and pageout walks

[Balbir Singh <balbirs@nvidia.com>]

On Tue, Jul 07, 2026 at 06:45:08AM -0700, Usama Arif wrote:
> madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
> pmd_is_huge() check returns true for a device-private PMD. The subsequent
> !pmd_present() branch has a VM_BUG_ON() asserting migration is the only
> allowed non-present case; a device-private PMD trips it.
> 
> Potential trigger: an HMM-based GPU driver races with
> madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
> migrate_vma_pages() flips the PMD to a device-private entry before the
> PMD lock is acquired.
> 
> Skip device-private PMDs after taking the lock, before the !pmd_present()
> check.
> 
> Reported-by: sashiko-bot <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  mm/madvise.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 9292f60b19aa..870be398c6f3 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -384,6 +384,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
>  			return 0;
>  
>  		orig_pmd = *pmd;
> +		if (pmd_is_device_private_entry(orig_pmd))
> +			goto huge_unlock;
> +
>  		if (is_huge_zero_pmd(orig_pmd))
>  			goto huge_unlock;
>  
>

Reviewed-by: Balbir Singh <balbirs@nvidia.com>