[PATCH v2] mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs

[PATCH v2] mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs

[Baolin Wang]

Lance reported that when nothing else causes the mm to be considered for
khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger
scanning.

After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into
shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads
vma->flags directly.  However, when MADV_HUGEPAGE is handled,
khugepaged_enter_vma() is called before the VMA's flags have been updated,
so the check uses stale flags and incorrectly rejects the VMA for collapse.
As a result, khugepaged does not collapse the tmpfs file into PMD order in time.

Fix this by calling khugepaged_enter_vma() with the new VMA flags in
madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma()
in hugepage_madvise().

Reported-by: Lance Yang <lance.yang@linux.dev>
Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/
Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
Cc: stable@vger.kernel.org
Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
Changes from v1:
v1: https://lore.kernel.org/all/ed34ca03ae7d65e89467fb87bc961f5497049c00.1786948410.git.baolin.wang@linux.alibaba.com/
 - Update the commit message (per Lorenzo).
 - Call khugepaged_enter_vma() in madvise_update_vma() (per Lorenzo).
---
 mm/khugepaged.c | 6 ------
 mm/madvise.c    | 8 ++++++++
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 5a06e3942e88..79effd3f3da4 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
 	case MADV_HUGEPAGE:
 		*vm_flags &= ~VM_NOHUGEPAGE;
 		*vm_flags |= VM_HUGEPAGE;
-		/*
-		 * If the vma become good for khugepaged to scan,
-		 * register it here without waiting a page fault that
-		 * may not happen any time soon.
-		 */
-		khugepaged_enter_vma(vma, *vm_flags);
 		break;
 	case MADV_NOHUGEPAGE:
 		*vm_flags &= ~VM_HUGEPAGE;
diff --git a/mm/madvise.c b/mm/madvise.c
index c179938097bf..cb93cf82d8df 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags,
 	/* vm_flags is protected by the mmap_lock held in write mode. */
 	vma_start_write(vma);
 	vma->flags = new_vma_flags;
+	/*
+	 * If the vma become good for khugepaged to scan,
+	 * register it here without waiting a page fault that
+	 * may not happen any time soon.
+	 */
+	if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT))
+		khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags));
+
 	if (set_new_anon_name)
 		return replace_anon_vma_name(vma, anon_name);
 
-- 
2.47.3

Re: [PATCH v2] mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs

[Zi Yan]

On Mon Aug 17, 2026 at 10:47 PM EDT, Baolin Wang wrote:
> Lance reported that when nothing else causes the mm to be considered for
> khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger
> scanning.
>
> After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into
> shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads
> vma->flags directly.  However, when MADV_HUGEPAGE is handled,
> khugepaged_enter_vma() is called before the VMA's flags have been updated,
> so the check uses stale flags and incorrectly rejects the VMA for collapse.
> As a result, khugepaged does not collapse the tmpfs file into PMD order in time.
>
> Fix this by calling khugepaged_enter_vma() with the new VMA flags in
> madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma()
> in hugepage_madvise().
>
> Reported-by: Lance Yang <lance.yang@linux.dev>
> Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
> Cc: stable@vger.kernel.org
> Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> Changes from v1:
> v1: https://lore.kernel.org/all/ed34ca03ae7d65e89467fb87bc961f5497049c00.1786948410.git.baolin.wang@linux.alibaba.com/
>  - Update the commit message (per Lorenzo).
>  - Call khugepaged_enter_vma() in madvise_update_vma() (per Lorenzo).
> ---
>  mm/khugepaged.c | 6 ------
>  mm/madvise.c    | 8 ++++++++
>  2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 5a06e3942e88..79effd3f3da4 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
>  	case MADV_HUGEPAGE:
>  		*vm_flags &= ~VM_NOHUGEPAGE;
>  		*vm_flags |= VM_HUGEPAGE;
> -		/*
> -		 * If the vma become good for khugepaged to scan,
> -		 * register it here without waiting a page fault that
> -		 * may not happen any time soon.
> -		 */
> -		khugepaged_enter_vma(vma, *vm_flags);
>  		break;
>  	case MADV_NOHUGEPAGE:
>  		*vm_flags &= ~VM_HUGEPAGE;
> diff --git a/mm/madvise.c b/mm/madvise.c
> index c179938097bf..cb93cf82d8df 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags,
>  	/* vm_flags is protected by the mmap_lock held in write mode. */
>  	vma_start_write(vma);
>  	vma->flags = new_vma_flags;
> +	/*
> +	 * If the vma become good for khugepaged to scan,
> +	 * register it here without waiting a page fault that
> +	 * may not happen any time soon.
> +	 */
> +	if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT))
> +		khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags));

LGTM.

Reviewed-by: Zi Yan <ziy@nvidia.com>

-- 
Best Regards,
Yan, Zi

Re: [PATCH v2] mm: fix incorrect vm_flags usage when checking allowable orders for tmpfs

[Lorenzo Stoakes (ARM)]

On Tue, Aug 18, 2026 at 10:47:26AM +0800, Baolin Wang wrote:
> Lance reported that when nothing else causes the mm to be considered for
> khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger
> scanning.
>
> After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into
> shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads
> vma->flags directly.  However, when MADV_HUGEPAGE is handled,
> khugepaged_enter_vma() is called before the VMA's flags have been updated,
> so the check uses stale flags and incorrectly rejects the VMA for collapse.
> As a result, khugepaged does not collapse the tmpfs file into PMD order in time.
>
> Fix this by calling khugepaged_enter_vma() with the new VMA flags in
> madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma()
> in hugepage_madvise().
>
> Reported-by: Lance Yang <lance.yang@linux.dev>
> Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
> Cc: stable@vger.kernel.org
> Suggested-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Thanks LGTM so:

Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

> ---
> Changes from v1:
> v1: https://lore.kernel.org/all/ed34ca03ae7d65e89467fb87bc961f5497049c00.1786948410.git.baolin.wang@linux.alibaba.com/
>  - Update the commit message (per Lorenzo).
>  - Call khugepaged_enter_vma() in madvise_update_vma() (per Lorenzo).
> ---
>  mm/khugepaged.c | 6 ------
>  mm/madvise.c    | 8 ++++++++
>  2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 5a06e3942e88..79effd3f3da4 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
>  	case MADV_HUGEPAGE:
>  		*vm_flags &= ~VM_NOHUGEPAGE;
>  		*vm_flags |= VM_HUGEPAGE;
> -		/*
> -		 * If the vma become good for khugepaged to scan,
> -		 * register it here without waiting a page fault that
> -		 * may not happen any time soon.
> -		 */
> -		khugepaged_enter_vma(vma, *vm_flags);
>  		break;
>  	case MADV_NOHUGEPAGE:
>  		*vm_flags &= ~VM_HUGEPAGE;
> diff --git a/mm/madvise.c b/mm/madvise.c
> index c179938097bf..cb93cf82d8df 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags,
>  	/* vm_flags is protected by the mmap_lock held in write mode. */
>  	vma_start_write(vma);
>  	vma->flags = new_vma_flags;
> +	/*
> +	 * If the vma become good for khugepaged to scan,
> +	 * register it here without waiting a page fault that
> +	 * may not happen any time soon.
> +	 */
> +	if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT))
> +		khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags));
> +
>  	if (set_new_anon_name)
>  		return replace_anon_vma_name(vma, anon_name);
>
> --
> 2.47.3
>

--
Cheers, Lorenzo