[PATCH] mm: shmem: ignore sysfs configs for shmem forced collapse

[PATCH] mm: shmem: ignore sysfs configs for shmem forced collapse

[Baolin Wang]

According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
ignore any THP or mTHP interface settings.

However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
(although that is the default), which can cause MADV_COLLAPSE to fail
unexpectedly.

Fix this by returning the result of shmem_huge_global_enabled() directly
when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
shmem THP/mTHP settings.

Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
Hi Kiryl,

I did not update the selftests, as I hope this fix will be merged first.
After this fix has landed, I can send a patch to update the selftests.
---
 mm/shmem.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index c92ed17dbc4a..b572c60f2af8 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2049,8 +2049,11 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
 
 	global_orders = shmem_huge_global_enabled(inode, index, write_end,
 						  shmem_huge_force, vma, vm_flags);
-	/* Tmpfs huge pages allocation */
-	if (!vma || !vma_is_anon_shmem(vma))
+	/*
+	 * Tmpfs huge pages allocation or forced collapse ignores
+	 * sysfs configs.
+	 */
+	if (!vma || !vma_is_anon_shmem(vma) || shmem_huge_force)
 		return global_orders;
 
 	/*
-- 
2.47.3

Re: [PATCH] mm: shmem: ignore sysfs configs for shmem forced collapse

[Kiryl Shutsemau]

On Fri, Sep 11, 2026 at 12:04:33PM +0800, Baolin Wang wrote:
> According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
> ignore any THP or mTHP interface settings.
> 
> However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
> into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
> will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
> (although that is the default), which can cause MADV_COLLAPSE to fail
> unexpectedly.
> 
> Fix this by returning the result of shmem_huge_global_enabled() directly
> when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
> shmem THP/mTHP settings.
> 
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")

Hm. I don't think it is the right Fixes tag. "No functional changes" in
its commit message seems to be true.

This might be a better fit:

Fixes: 26c7d8413aaf ("mm: thp: support "THPeligible" semantics for mTHP with anonymous shmem")

> Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Thanks!

Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>


-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] mm: shmem: ignore sysfs configs for shmem forced collapse

[Zi Yan]

On 11 Sep 2026, at 0:04, Baolin Wang wrote:

> According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
> ignore any THP or mTHP interface settings.
>
> However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
> into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
> will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
> (although that is the default), which can cause MADV_COLLAPSE to fail
> unexpectedly.
>
> Fix this by returning the result of shmem_huge_global_enabled() directly
> when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
> shmem THP/mTHP settings.
>
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
> Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> Hi Kiryl,
>
> I did not update the selftests, as I hope this fix will be merged first.
> After this fix has landed, I can send a patch to update the selftests.
> ---
>  mm/shmem.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
Makes sense.

Acked-by: Zi Yan <ziy@nvidia.com>

Best Regards,
Yan, Zi

Re: [PATCH] mm: shmem: ignore sysfs configs for shmem forced collapse

[Lance Yang]

On Fri, Sep 11, 2026 at 12:04:33PM +0800, Baolin Wang wrote:
>According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
>ignore any THP or mTHP interface settings.
>
>However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
>into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
>will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
>(although that is the default), which can cause MADV_COLLAPSE to fail
>unexpectedly.
>
>Fix this by returning the result of shmem_huge_global_enabled() directly
>when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
>shmem THP/mTHP settings.
>
>Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
>Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
>Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
>Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>---

With the Fixes tag corrected as Kiryl suggested, LGTM!

Reviewed-by: Lance Yang <lance.yang@linux.dev>