Re: [PATCH v2] mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs

["Zi Yan" <ziy@nvidia.com>]

On Mon Aug 17, 2026 at 10:47 PM EDT, Baolin Wang wrote:
> Lance reported that when nothing else causes the mm to be considered for
> khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger
> scanning.
>
> After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into
> shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads
> vma->flags directly.  However, when MADV_HUGEPAGE is handled,
> khugepaged_enter_vma() is called before the VMA's flags have been updated,
> so the check uses stale flags and incorrectly rejects the VMA for collapse.
> As a result, khugepaged does not collapse the tmpfs file into PMD order in time.
>
> Fix this by calling khugepaged_enter_vma() with the new VMA flags in
> madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma()
> in hugepage_madvise().
>
> Reported-by: Lance Yang <lance.yang@linux.dev>
> Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
> Cc: stable@vger.kernel.org
> Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> Changes from v1:
> v1: https://lore.kernel.org/all/ed34ca03ae7d65e89467fb87bc961f5497049c00.1786948410.git.baolin.wang@linux.alibaba.com/
>  - Update the commit message (per Lorenzo).
>  - Call khugepaged_enter_vma() in madvise_update_vma() (per Lorenzo).
> ---
>  mm/khugepaged.c | 6 ------
>  mm/madvise.c    | 8 ++++++++
>  2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 5a06e3942e88..79effd3f3da4 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
>  	case MADV_HUGEPAGE:
>  		*vm_flags &= ~VM_NOHUGEPAGE;
>  		*vm_flags |= VM_HUGEPAGE;
> -		/*
> -		 * If the vma become good for khugepaged to scan,
> -		 * register it here without waiting a page fault that
> -		 * may not happen any time soon.
> -		 */
> -		khugepaged_enter_vma(vma, *vm_flags);
>  		break;
>  	case MADV_NOHUGEPAGE:
>  		*vm_flags &= ~VM_HUGEPAGE;
> diff --git a/mm/madvise.c b/mm/madvise.c
> index c179938097bf..cb93cf82d8df 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags,
>  	/* vm_flags is protected by the mmap_lock held in write mode. */
>  	vma_start_write(vma);
>  	vma->flags = new_vma_flags;
> +	/*
> +	 * If the vma become good for khugepaged to scan,
> +	 * register it here without waiting a page fault that
> +	 * may not happen any time soon.
> +	 */
> +	if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT))
> +		khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags));

LGTM.

Reviewed-by: Zi Yan <ziy@nvidia.com>

-- 
Best Regards,
Yan, Zi