[PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

[Seunghyun An <san55@wisc.edu>]

__vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt
cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track
MAP_PRIVATE file-backed anon folios"), which made anon rmap use it,
__folio_set_anon() reads it on every new anonymous folio, i.e. once per
fault that maps a new anonymous page, between vma_start_read() and
vma_end_read().

When several threads fault on the same VMA, the other CPUs keep writing
vm_refcnt from vma_start_read()/vma_end_read(), so this read often
misses and pulls the line over. The extra load can force a coherence
transfer and makes the following vm_refcnt updates more expensive.

For anonymous VMAs the anonymous page offset is always equal to vm_pgoff
(linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM;
it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f
("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")),
so compute the index from vm_pgoff as before. That brings back the
pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still
read the split field. Keep the assertion, the other way round.

Moving __vm_anon_pgoff_hi out of the line does not work without growing
vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly
192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields
that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi
halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(),
so it would just move the miss.

Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
mainline after v7.4-rc1 (the base of this series). Threads share one
anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
faults/s, median of 3:

  CPUs                      base      patched
  1 CCD,   7 threads        2.360M    2.369M   +0.4%
  7 CCDs,  1 thread each    1.738M    1.982M  +14.0%
  12 CCDs, 1 thread each    2.288M    2.548M  +11.4%

On 7.3.1 (with the 7.3 variant of this change) the same runs gave +1.9%,
+16.0% and +29.4%. There, perf c2c shows the folio_add_new_anon_rmap()
load at offset 132 disappearing from the vm_refcnt line and the average
load latency of the vm_refcnt inc/dec on that line dropping by about
45%. With one VMA per thread, or with processes, this patch alone
changes throughput by -2.0% to -0.1% on that base (less than 1% on 7.3.1);
the next patch more than recovers that.

will-it-scale page_fault1 threads does not show the difference on this
machine: each thread mmap()s and munmap()s its own region, which merges
VMAs and is dominated by VMA write locking, and its results vary by up
to 10% between boots of the same kernel.

This relies on the /dev/zero commit above, which is not in v7.3, so this
is not tagged for stable: without it, MAP_PRIVATE /dev/zero VMAs are
anonymous but keep the file offset in vm_pgoff, and must keep using the
anon pgoff field.

The tools/testing/vma copy of the helper is updated to match.

Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 include/linux/pagemap.h         | 19 ++++++++++++++-----
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 2 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 73af18a373..300ee83fc1 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1122,13 +1122,22 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	/*
+	 * For anonymous VMAs the anonymous page offset always equals vm_pgoff,
+	 * so use it. This keeps the anonymous fault path (__folio_set_anon())
+	 * off __vm_anon_pgoff_hi, which shares a cacheline with vm_refcnt and
+	 * is therefore contended by concurrent faults on the same VMA.
+	 */
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 struct wait_page_key {
diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h
index dc4927d262..f3d69b0c74 100644
--- a/tools/testing/vma/include/dup.h
+++ b/tools/testing/vma/include/dup.h
@@ -1655,13 +1655,16 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 extern const struct file_operations zero_fops;
-- 
2.53.0