* [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER
@ 2026-11-17 12:16 Jingyi Sun
2026-11-18 2:31 ` Baolin Wang
2026-11-18 2:40 ` Luiz Capitulino (Red Hat)
0 siblings, 2 replies; 3+ messages in thread
From: Jingyi Sun @ 2026-11-17 12:16 UTC (permalink / raw)
To: Hugh Dickins, David Hildenbrand, Andrew Morton
Cc: Baolin Wang, linux-mm, linux-kernel, stable, Luiz Capitulino
When the top-level shmem_enabled knob is set to "force",
shmem_allowable_huge_orders() returns huge_shmem_orders_inherit for
anonymous shmem without masking it with THP_ORDERS_ALL_FILE_DEFAULT,
unlike the other return path at the end of the same function.
huge_shmem_orders_inherit defaults to BIT(HPAGE_PMD_ORDER). On arm64
with 64K base pages HPAGE_PMD_ORDER is 13, while MAX_PAGECACHE_ORDER
is 11 because of the xarray split limit. With shmem_enabled=force, an
anonymous MAP_SHARED mapping can therefore get a 512M (order-13) folio
in the page cache. Splitting it later, e.g. on swap-out from
shmem_writeout(), triggers
WARNING: lib/xarray.c:1051 at xas_split_alloc
and the split fails, so the folio can never be swapped out.
This is the same problem that commit 9fd154ba926b ("mm/shmem: disable
PMD-sized page cache if needed") and commit b66b1b71d7ff ("mm: shmem:
avoid allocating huge pages larger than MAX_PAGECACHE_ORDER for shmem")
fixed for the other paths; the force branch was left out. Mask it with
THP_ORDERS_ALL_FILE_DEFAULT as well.
Reproduced on v7.3-rc2 in QEMU (arm64, 64K pages, -cpu cortex-a72,
4G RAM, zram swap): with shmem_enabled=force, a PMD-aligned
MAP_SHARED|MAP_ANONYMOUS mapping of one PMD size got a single
PMD-mapped folio (ShmemPmdMapped: 524288 kB) and MADV_PAGEOUT
triggered the warning above in 3/3 runs. With this change: 0/3.
shmem_enabled=always and 4K pages did not show the warning. The code is
unchanged in v7.4-rc3 and in mm-unstable, but the test has not been
re-run on those trees.
The issue was found with a research static analyzer for
cross-architecture dependency bugs and confirmed with the QEMU
reproducer described above.
Fixes: e7a2ab7b3bb5 ("mm: shmem: add mTHP support for anonymous shmem")
Cc: stable@vger.kernel.org
Cc: Luiz Capitulino <luizcap@redhat.com>
Assisted-by: LLM
Signed-off-by: Jingyi Sun <1045704958@qq.com>
---
Note: this touches the same line as "mm: thp: always enable mTHP
support" (Luiz Capitulino, v8 14/14), which is still under review. That
change masks the force branch with ~disabled_orders but does not add
THP_ORDERS_ALL_FILE_DEFAULT either, so it does not address this issue.
Happy to rebase on top of that series if it lands first.
Link: https://lore.kernel.org/all/752f528f0fed5cdc9de12b54260b0495d4e5a6cb.1789695931.git.luizcap@redhat.com/
mm/shmem.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 10cef89b00b0..41bb3f2c6b63 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2114,7 +2114,8 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
* means non-PMD sized THP can not override 'huge' mount option now.
*/
if (shmem_huge == SHMEM_HUGE_FORCE)
- return READ_ONCE(huge_shmem_orders_inherit);
+ return THP_ORDERS_ALL_FILE_DEFAULT &
+ READ_ONCE(huge_shmem_orders_inherit);
/* Allow mTHP that will be fully within i_size. */
mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0);
base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
--
2.43.0
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER
2026-11-17 12:16 [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER Jingyi Sun
@ 2026-11-18 2:31 ` Baolin Wang
2026-11-18 2:40 ` Luiz Capitulino (Red Hat)
1 sibling, 0 replies; 3+ messages in thread
From: Baolin Wang @ 2026-11-18 2:31 UTC (permalink / raw)
To: Jingyi Sun, Hugh Dickins, David Hildenbrand, Andrew Morton
Cc: linux-mm, linux-kernel, stable, Luiz Capitulino
On 11/17/26 8:16 PM, Jingyi Sun wrote:
> When the top-level shmem_enabled knob is set to "force",
> shmem_allowable_huge_orders() returns huge_shmem_orders_inherit for
> anonymous shmem without masking it with THP_ORDERS_ALL_FILE_DEFAULT,
> unlike the other return path at the end of the same function.
>
> huge_shmem_orders_inherit defaults to BIT(HPAGE_PMD_ORDER). On arm64
> with 64K base pages HPAGE_PMD_ORDER is 13, while MAX_PAGECACHE_ORDER
> is 11 because of the xarray split limit. With shmem_enabled=force, an
> anonymous MAP_SHARED mapping can therefore get a 512M (order-13) folio
> in the page cache. Splitting it later, e.g. on swap-out from
> shmem_writeout(), triggers
>
> WARNING: lib/xarray.c:1051 at xas_split_alloc
>
> and the split fails, so the folio can never be swapped out.
>
> This is the same problem that commit 9fd154ba926b ("mm/shmem: disable
> PMD-sized page cache if needed") and commit b66b1b71d7ff ("mm: shmem:
> avoid allocating huge pages larger than MAX_PAGECACHE_ORDER for shmem")
> fixed for the other paths; the force branch was left out. Mask it with
> THP_ORDERS_ALL_FILE_DEFAULT as well.
>
> Reproduced on v7.3-rc2 in QEMU (arm64, 64K pages, -cpu cortex-a72,
> 4G RAM, zram swap): with shmem_enabled=force, a PMD-aligned
> MAP_SHARED|MAP_ANONYMOUS mapping of one PMD size got a single
> PMD-mapped folio (ShmemPmdMapped: 524288 kB) and MADV_PAGEOUT
> triggered the warning above in 3/3 runs. With this change: 0/3.
> shmem_enabled=always and 4K pages did not show the warning. The code is
> unchanged in v7.4-rc3 and in mm-unstable, but the test has not been
> re-run on those trees.
>
> The issue was found with a research static analyzer for
> cross-architecture dependency bugs and confirmed with the QEMU
> reproducer described above.
>
> Fixes: e7a2ab7b3bb5 ("mm: shmem: add mTHP support for anonymous shmem")
> Cc: stable@vger.kernel.org
> Cc: Luiz Capitulino <luizcap@redhat.com>
> Assisted-by: LLM
Please drop this tag.
> Signed-off-by: Jingyi Sun <1045704958@qq.com>
> ---
LGTM.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER
2026-11-17 12:16 [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER Jingyi Sun
2026-11-18 2:31 ` Baolin Wang
@ 2026-11-18 2:40 ` Luiz Capitulino (Red Hat)
1 sibling, 0 replies; 3+ messages in thread
From: Luiz Capitulino (Red Hat) @ 2026-11-18 2:40 UTC (permalink / raw)
To: Jingyi Sun, Hugh Dickins, David Hildenbrand, Andrew Morton
Cc: Baolin Wang, linux-mm, linux-kernel, stable
On 11/17/26 7:16 AM, Jingyi Sun wrote:
> When the top-level shmem_enabled knob is set to "force",
> shmem_allowable_huge_orders() returns huge_shmem_orders_inherit for
> anonymous shmem without masking it with THP_ORDERS_ALL_FILE_DEFAULT,
> unlike the other return path at the end of the same function.
>
> huge_shmem_orders_inherit defaults to BIT(HPAGE_PMD_ORDER). On arm64
> with 64K base pages HPAGE_PMD_ORDER is 13, while MAX_PAGECACHE_ORDER
> is 11 because of the xarray split limit. With shmem_enabled=force, an
> anonymous MAP_SHARED mapping can therefore get a 512M (order-13) folio
> in the page cache. Splitting it later, e.g. on swap-out from
> shmem_writeout(), triggers
>
> WARNING: lib/xarray.c:1051 at xas_split_alloc
>
> and the split fails, so the folio can never be swapped out.
>
> This is the same problem that commit 9fd154ba926b ("mm/shmem: disable
> PMD-sized page cache if needed") and commit b66b1b71d7ff ("mm: shmem:
> avoid allocating huge pages larger than MAX_PAGECACHE_ORDER for shmem")
> fixed for the other paths; the force branch was left out. Mask it with
> THP_ORDERS_ALL_FILE_DEFAULT as well.
>
> Reproduced on v7.3-rc2 in QEMU (arm64, 64K pages, -cpu cortex-a72,
> 4G RAM, zram swap): with shmem_enabled=force, a PMD-aligned
> MAP_SHARED|MAP_ANONYMOUS mapping of one PMD size got a single
> PMD-mapped folio (ShmemPmdMapped: 524288 kB) and MADV_PAGEOUT
> triggered the warning above in 3/3 runs. With this change: 0/3.
> shmem_enabled=always and 4K pages did not show the warning. The code is
> unchanged in v7.4-rc3 and in mm-unstable, but the test has not been
> re-run on those trees.
>
> The issue was found with a research static analyzer for
> cross-architecture dependency bugs and confirmed with the QEMU
> reproducer described above.
>
> Fixes: e7a2ab7b3bb5 ("mm: shmem: add mTHP support for anonymous shmem")
> Cc: stable@vger.kernel.org
> Cc: Luiz Capitulino <luizcap@redhat.com>
> Assisted-by: LLM
> Signed-off-by: Jingyi Sun <1045704958@qq.com>
> ---
> Note: this touches the same line as "mm: thp: always enable mTHP
> support" (Luiz Capitulino, v8 14/14), which is still under review. That
> change masks the force branch with ~disabled_orders but does not add
> THP_ORDERS_ALL_FILE_DEFAULT either, so it does not address this issue.
> Happy to rebase on top of that series if it lands first.
I'm late with v9 so no way my series will land before this patch, but
thanks for the consideration :)
>
> Link: https://lore.kernel.org/all/752f528f0fed5cdc9de12b54260b0495d4e5a6cb.1789695931.git.luizcap@redhat.com/
>
> mm/shmem.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 10cef89b00b0..41bb3f2c6b63 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -2114,7 +2114,8 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
> * means non-PMD sized THP can not override 'huge' mount option now.
> */
> if (shmem_huge == SHMEM_HUGE_FORCE)
> - return READ_ONCE(huge_shmem_orders_inherit);
> + return THP_ORDERS_ALL_FILE_DEFAULT &
> + READ_ONCE(huge_shmem_orders_inherit);
>
> /* Allow mTHP that will be fully within i_size. */
> mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0);
>
> base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-18 2:40 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 12:16 [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER Jingyi Sun
2026-11-18 2:31 ` Baolin Wang
2026-11-18 2:40 ` Luiz Capitulino (Red Hat)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox