Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1
@ 2026-11-17 19:32 Samridh via B4 Relay
  2026-11-18  3:29 ` Baolin Wang
  2026-11-18 19:32 ` sashiko-bot
  0 siblings, 2 replies; 4+ messages in thread
From: Samridh via B4 Relay @ 2026-11-17 19:32 UTC (permalink / raw)
  To: Hugh Dickins, David Hildenbrand, Baolin Wang, Andrew Morton
  Cc: linux-mm, linux-kernel, Samridh

From: Samridh <samridh@modal.com>

shmem_get_folio_gfp() clears a new folio before marking it uptodate.
With init_on_alloc=1 the page allocator has already zeroed it, so the
folio ends up being zeroed twice.

For SGP_CACHE, the only mode that clears a new folio, ask the page
allocator for a zeroed folio with __GFP_ZERO. Folios that are already in
the page cache but not uptodate, such as those left by fallocate(), are
still cleared. With init_on_alloc=0 the zeroing moves into the
allocator, so a folio is zeroed for nothing when the memcg charge, the
page cache insert or the block accounting fails.

Time to write one byte to each page of a new 4 GiB memfd mapping, with
shmem_enabled=always and init_on_alloc=1 (median of 24 runs on 6.18.55,
x86_64 VMs):

                  before   after
  Xeon 6975P-C    0.620s   0.416s
  EPYC 9R14       0.216s   0.158s

With 4K folios, or with init_on_alloc=0, the times change by 3% or
less. We found this while running gVisor, which keeps sandbox memory in
a memfd.

I wrote this patch with help from an AI coding assistant (Claude Code),
then reviewed and tested it.

Signed-off-by: Samridh <samridh@modal.com>
---
Tested on 6.18.55 with this change backported, on Intel and AMD x86_64
VMs, and build-tested on mm/core for-next. I also wrote a simple test.
It fills memory with data, frees it, and then checks that new memfd
pages read as zero. It covers MADV_HUGEPAGE and MADV_NOHUGEPAGE on the
fault and fallocate paths. It passes on both machines with
init_on_alloc/init_on_free at 1/0, 0/0 and 1/1.

The idea came from David's comment on the __GFP_PREZEROED RFC [1].
One thing I'm not sure about is whether flush_dcache_folio() after the
allocator has zeroed the folio is enough on architectures with cache
aliasing, so a second look there would be welcome.

[1] https://lore.kernel.org/all/b258160c-5287-43dd-b56e-268c403b85ea@kernel.org/
---
 mm/shmem.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 5cb628cc9c6b..d2c6781e86a5 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2692,6 +2692,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	int error;
 	bool alloced;
 	unsigned long orders = 0;
+	gfp_t zero_gfp = sgp == SGP_CACHE ? __GFP_ZERO : 0;
 
 	if (WARN_ON_ONCE(!shmem_mapping(inode->i_mapping)))
 		return -EINVAL;
@@ -2770,7 +2771,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 
 		huge_gfp = vma_thp_gfp_mask(vma);
 		huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
-		folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
+		folio = shmem_alloc_and_add_folio(vmf, huge_gfp | zero_gfp,
 				inode, index, fault_mm, orders);
 		if (!IS_ERR(folio)) {
 			if (folio_test_pmd_mappable(folio))
@@ -2782,7 +2783,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 			goto repeat;
 	}
 
-	folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, 0);
+	folio = shmem_alloc_and_add_folio(vmf, gfp | zero_gfp, inode, index,
+					  fault_mm, 0);
 	if (IS_ERR(folio)) {
 		error = PTR_ERR(folio);
 		if (error == -EEXIST)
@@ -2810,8 +2812,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) {
 		long i, n = folio_nr_pages(folio);
 
-		for (i = 0; i < n; i++)
-			clear_highpage(folio_page(folio, i));
+		/* A folio allocated with __GFP_ZERO is already zeroed. */
+		if (!alloced || !zero_gfp) {
+			for (i = 0; i < n; i++)
+				clear_highpage(folio_page(folio, i));
+		}
 		flush_dcache_folio(folio);
 		folio_mark_uptodate(folio);
 	}

---
base-commit: 0c524518413094607a6178c53a65815762bc9c45
change-id: 20261117-samridh-shmem-send-9fd2d8231e8e

Best regards,
--  
Samridh <samridh@modal.com>




^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1
  2026-11-17 19:32 [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1 Samridh via B4 Relay
@ 2026-11-18  3:29 ` Baolin Wang
  2026-11-18  4:59   ` Fnu Samridh
  2026-11-18 19:32 ` sashiko-bot
  1 sibling, 1 reply; 4+ messages in thread
From: Baolin Wang @ 2026-11-18  3:29 UTC (permalink / raw)
  To: samridh, Hugh Dickins, David Hildenbrand, Andrew Morton
  Cc: linux-mm, linux-kernel



On 11/18/26 3:32 AM, Samridh via B4 Relay wrote:
> From: Samridh <samridh@modal.com>
> 
> shmem_get_folio_gfp() clears a new folio before marking it uptodate.
> With init_on_alloc=1 the page allocator has already zeroed it, so the
> folio ends up being zeroed twice.
> 
> For SGP_CACHE, the only mode that clears a new folio, ask the page
> allocator for a zeroed folio with __GFP_ZERO. Folios that are already in
> the page cache but not uptodate, such as those left by fallocate(), are
> still cleared. With init_on_alloc=0 the zeroing moves into the
> allocator, so a folio is zeroed for nothing when the memcg charge, the
> page cache insert or the block accounting fails.
> 
> Time to write one byte to each page of a new 4 GiB memfd mapping, with
> shmem_enabled=always and init_on_alloc=1 (median of 24 runs on 6.18.55,
> x86_64 VMs):
> 
>                    before   after
>    Xeon 6975P-C    0.620s   0.416s
>    EPYC 9R14       0.216s   0.158s
> 
> With 4K folios, or with init_on_alloc=0, the times change by 3% or
> less. We found this while running gVisor, which keeps sandbox memory in
> a memfd.
> 
> I wrote this patch with help from an AI coding assistant (Claude Code),
> then reviewed and tested it.
> 
> Signed-off-by: Samridh <samridh@modal.com>
> ---
> Tested on 6.18.55 with this change backported, on Intel and AMD x86_64
> VMs, and build-tested on mm/core for-next. I also wrote a simple test.
> It fills memory with data, frees it, and then checks that new memfd
> pages read as zero. It covers MADV_HUGEPAGE and MADV_NOHUGEPAGE on the
> fault and fallocate paths. It passes on both machines with
> init_on_alloc/init_on_free at 1/0, 0/0 and 1/1.
> 
> The idea came from David's comment on the __GFP_PREZEROED RFC [1].
> One thing I'm not sure about is whether flush_dcache_folio() after the
> allocator has zeroed the folio is enough on architectures with cache
> aliasing, so a second look there would be welcome.
> 
> [1] https://lore.kernel.org/all/b258160c-5287-43dd-b56e-268c403b85ea@kernel.org/
> ---
>   mm/shmem.c | 13 +++++++++----
>   1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 5cb628cc9c6b..d2c6781e86a5 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -2692,6 +2692,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>   	int error;
>   	bool alloced;
>   	unsigned long orders = 0;
> +	gfp_t zero_gfp = sgp == SGP_CACHE ? __GFP_ZERO : 0;
>   
>   	if (WARN_ON_ONCE(!shmem_mapping(inode->i_mapping)))
>   		return -EINVAL;
> @@ -2770,7 +2771,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>   
>   		huge_gfp = vma_thp_gfp_mask(vma);
>   		huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
> -		folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
> +		folio = shmem_alloc_and_add_folio(vmf, huge_gfp | zero_gfp,
>   				inode, index, fault_mm, orders);
>   		if (!IS_ERR(folio)) {
>   			if (folio_test_pmd_mappable(folio))
> @@ -2782,7 +2783,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>   			goto repeat;
>   	}
>   
> -	folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, 0);
> +	folio = shmem_alloc_and_add_folio(vmf, gfp | zero_gfp, inode, index,
> +					  fault_mm, 0);
>   	if (IS_ERR(folio)) {
>   		error = PTR_ERR(folio);
>   		if (error == -EEXIST)
> @@ -2810,8 +2812,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>   	if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) {
>   		long i, n = folio_nr_pages(folio);
>   
> -		for (i = 0; i < n; i++)
> -			clear_highpage(folio_page(folio, i));
> +		/* A folio allocated with __GFP_ZERO is already zeroed. */
> +		if (!alloced || !zero_gfp) {

I don't like the GFP flag hack. Why not use user_alloc_needs_zeroing() 
to decide whether we need to zero folios, like anonymous memory does?


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1
  2026-11-18  3:29 ` Baolin Wang
@ 2026-11-18  4:59   ` Fnu Samridh
  0 siblings, 0 replies; 4+ messages in thread
From: Fnu Samridh @ 2026-11-18  4:59 UTC (permalink / raw)
  To: Baolin Wang
  Cc: Hugh Dickins, David Hildenbrand, Andrew Morton, linux-mm,
	linux-kernel

On Tue, Nov 17, 2026 at 10:29 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
> I don't like the GFP flag hack. Why not use user_alloc_needs_zeroing()
> to decide whether we need to zero folios, like anonymous memory does?

I started with user_alloc_needs_zeroing(), but switched after a comment
on the list, thinking the helper wouldn't be accepted here. My mistake.
I'll go back to it in v2, which also takes care of my flush_dcache_folio()
question. Thanks!


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1
  2026-11-17 19:32 [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1 Samridh via B4 Relay
  2026-11-18  3:29 ` Baolin Wang
@ 2026-11-18 19:32 ` sashiko-bot
  1 sibling, 0 replies; 4+ messages in thread
From: sashiko-bot @ 2026-11-18 19:32 UTC (permalink / raw)
  To: Samridh, Hugh Dickins, David Hildenbrand, Baolin Wang,
	Andrew Morton
  Cc: linux-mm

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] mm/shmem: single-word Signed-off-by identity in commit message
- [High] mm/shmem: cross-CPU dcache flush after __GFP_ZERO in
  shmem_get_folio_gfp()
--

commit acaf023dbfc00405b46eac6010b09d293eda6ffd
Author: Samridh <samridh@modal.com>

> mm/shmem: avoid zeroing new folios twice with init_on_alloc=1
>
> shmem_get_folio_gfp() clears a new folio before marking it uptodate.
> With init_on_alloc=1 the page allocator has already zeroed it, so the
> folio ends up being zeroed twice.
>
> For SGP_CACHE, the only mode that clears a new folio, ask the page
> allocator for a zeroed folio with __GFP_ZERO. Folios that are already in
> the page cache but not uptodate, such as those left by fallocate(), are
> still cleared. With init_on_alloc=0 the zeroing moves into the
> allocator, so a folio is zeroed for nothing when the memcg charge, the
> page cache insert or the block accounting fails.
>
> Time to write one byte to each page of a new 4 GiB memfd mapping, with
> shmem_enabled=always and init_on_alloc=1 (median of 24 runs on 6.18.55,
> x86_64 VMs):
>
>                   before   after
>   Xeon 6975P-C    0.620s   0.416s
>   EPYC 9R14       0.216s   0.158s
>
> With 4K folios, or with init_on_alloc=0, the times change by 3% or
> less. We found this while running gVisor, which keeps sandbox memory in
> a memfd.
>
> I wrote this patch with help from an AI coding assistant (Claude Code),
> then reviewed and tested it.
>
> Signed-off-by: Samridh <samridh@modal.com>

[Severity: High]
Documentation/process/submitting-patches.rst requires a full real name in
the Author header and Signed-off-by: tag rather than a single-word name or
handle.

> diff --git a/mm/shmem.c b/mm/shmem.c
> index 5cb628cc9c6bb..d2c6781e86a55 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -2692,6 +2692,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>  	int error;
>  	bool alloced;
>  	unsigned long orders = 0;
> +	gfp_t zero_gfp = sgp == SGP_CACHE ? __GFP_ZERO : 0;
>  
>  	if (WARN_ON_ONCE(!shmem_mapping(inode->i_mapping)))
>  		return -EINVAL;
> @@ -2770,7 +2771,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>  
>  		huge_gfp = vma_thp_gfp_mask(vma);
>  		huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
> -		folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
> +		folio = shmem_alloc_and_add_folio(vmf, huge_gfp | zero_gfp,
>  				inode, index, fault_mm, orders);
>  		if (!IS_ERR(folio)) {
>  			if (folio_test_pmd_mappable(folio))
> @@ -2782,7 +2783,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>  			goto repeat;
>  	}
>  
> -	folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, 0);
> +	folio = shmem_alloc_and_add_folio(vmf, gfp | zero_gfp, inode, index,
> +					  fault_mm, 0);
>  	if (IS_ERR(folio)) {
>  		error = PTR_ERR(folio);
>  		if (error == -EEXIST)
> @@ -2810,8 +2812,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
>  	if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) {
>  		long i, n = folio_nr_pages(folio);
>  
> -		for (i = 0; i < n; i++)
> -			clear_highpage(folio_page(folio, i));
> +		/* A folio allocated with __GFP_ZERO is already zeroed. */
> +		if (!alloced || !zero_gfp) {
> +			for (i = 0; i < n; i++)
> +				clear_highpage(folio_page(folio, i));
> +		}
>  		flush_dcache_folio(folio);
>  		folio_mark_uptodate(folio);
>  	}

[Severity: High]
Can skipping clear_highpage() here cause flush_dcache_folio(folio) to run
on a different CPU from the one that zeroed the folio?

flush_dcache_folio() requires running on the CPU that performed the kernel
stores into the folio.  On sparc64 and on 32-bit SMP ARM (when
cache_ops_need_broadcast() is true), flush_dcache_folio() only flushes or
records the local CPU's D-cache:

arch/sparc/mm/init_64.c:flush_dcache_folio() {
	...
	this_cpu = get_cpu();

	mapping = folio_flush_mapping(folio);
	if (mapping && !mapping_mapped(mapping)) {
		bool dirty = test_bit(PG_dcache_dirty, &folio->flags.f);
		if (dirty) {
			int dirty_cpu = dcache_dirty_cpu(folio);

			if (dirty_cpu == this_cpu)
				goto out;
			smp_flush_dcache_folio_impl(folio, dirty_cpu);
		}
		set_dcache_dirty(folio, this_cpu);
	} else {
		flush_dcache_folio_impl(folio);
	}
	...
}

With __GFP_ZERO, the zeroing stores can happen on another CPU before
flush_dcache_folio(folio) is reached:

CPU0                              CPU1
shmem_get_folio_gfp()
  shmem_alloc_and_add_folio()
    shmem_alloc_folio(..., __GFP_ZERO)
      post_alloc_hook()
        kernel_init_pages()
    mem_cgroup_charge() (sleeps, task migrates)
                                  ...
                                  flush_dcache_folio(folio)
                                    /* flushes/records CPU1, leaving
                                     * zeroed lines dirty in CPU0's L1 */

In addition, when init_on_free=1, post_alloc_hook() skips zeroing on
allocation because the page was zeroed when freed in
__free_pages_prepare(), which may have happened on another CPU without a
D-cache flush.  In either case, a userspace mapping at an aliasing virtual
address can read stale non-zero data from memory instead of the zeroed
L1 cache lines.

Anonymous folio allocations avoid this by checking
user_alloc_needs_zeroing(), which keeps explicit post-allocation zeroing
when cpu_dcache_is_aliasing() || cpu_icache_is_aliasing() is true.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117-samridh-shmem-send-v1-1-1b897ff562d9@modal.com?part=1


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-18 19:32 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 19:32 [PATCH] mm/shmem: avoid zeroing new folios twice with init_on_alloc=1 Samridh via B4 Relay
2026-11-18  3:29 ` Baolin Wang
2026-11-18  4:59   ` Fnu Samridh
2026-11-18 19:32 ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox