Re: [PATCH v2 2/3] mm/madvise: skip device-private PMDs in cold and pageout walks

["David Hildenbrand (Arm)" <david@kernel.org>]

On 7/8/26 14:20, Usama Arif wrote:
> madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
> pmd_is_huge() check returns true for a device-private PMD. The subsequent
> !pmd_present() branch has a VM_BUG_ON() asserting migration is the only
> allowed non-present case; a device-private PMD trips it.
> 
> Allow device-private PMDs in that non-present assertion and continue to
> huge_unlock before calling pmd_folio(). This keeps the assertion for
> unexpected PMD softleafs while skipping device-private PMDs like other
> non-present PMDs in this path.
> 
> Potential trigger: an HMM-based GPU driver races with
> madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
> migrate_vma_pages() flips the PMD to a device-private entry before the
> PMD lock is acquired.
> 
> Reported-by: sashiko-bot <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
> Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
> Cc: <stable@vger.kernel.org>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Balbir Singh <balbirs@nvidia.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  mm/madvise.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 9292f60b19aa..1065b5a84ea7 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -389,7 +389,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
>  
>  		if (unlikely(!pmd_present(orig_pmd))) {
>  			VM_BUG_ON(thp_migration_supported() &&
> -					!pmd_is_migration_entry(orig_pmd));
> +					!pmd_is_migration_entry(orig_pmd) &&
> +					!pmd_is_device_private_entry(orig_pmd));

thp_migration_supported() is just effectively CONFIG_TRANSPARENT_HUGEPAGE &&
CONFIG_ARCH_HAS_PMD_SOFTLEAVES.

pmd_is_migration_entry() -> softleaf_is_migration(softleaf_from_pmd(pmd)) ==
false without CONFIG_ARCH_HAS_PMD_SOFTLEAVES. ... and
CONFIG_ARCH_HAS_PMD_SOFTLEAVES is only selected if TRANSPARENT_HUGEPAGE.

While at it, can you drop the thp_migration_supported() check and turn the
VM_BUG_ON into a VM_WARN_ON_ONCE() ?

-- 
Cheers,

David