[PATCH 2/2] mm/vma: only assert VMA stability in vma_has_anon_rmap() with CONFIG_DEBUG_VM

[Seunghyun An <san55@wisc.edu>]

vma_has_anon_rmap() calls vma_is_attached() and vma_assert_stabilised()
unconditionally. Without lockdep, vma_assert_stabilised() reads
mm->mmap_lock and then vma_assert_locked() reads vm_refcnt. These reads
stay in CONFIG_DEBUG_VM=n builds: refcount_read() and rwsem_is_locked()
are atomic reads (READ_ONCE()), which the compiler has to keep even
though the VM_WARN_ON_ONCE_VMA() checks consuming them compile away.

__vmf_anon_prepare() calls this on every anonymous fault that allocates
a new page or breaks CoW, so every such fault handled under the per-VMA
lock now loads vm_refcnt twice more, plus the mmap_lock count. When
threads fault on the same VMA,
other CPUs keep writing vm_refcnt in vma_start_read() and vma_end_read(),
so these loads can miss.

The assertion has no effect outside debug builds, so only do it with
CONFIG_DEBUG_VM. The fast path is then a plain vma->anon_vma test again.

On top of the previous patch, same setup as there (AMD EPYC 9634, threads
faulting on one shared anonymous VMA, faults/s, median of 3):

  CPUs                      before    after
  1 CCD,   7 threads        2.369M    2.425M   +2.4%
  7 CCDs,  1 thread each    1.982M    2.215M  +11.7%
  12 CCDs, 1 thread each    2.548M    2.949M  +15.7%

With one VMA per thread, or with processes, the change is +0.3% to
+1.9% (the loads are gone from the local path as well). perf c2c no
longer shows __vmf_anon_prepare() on the vm_refcnt line.

Fixes: 38564710f443 ("mm: implement and use vma_has_anon_rmap(), silence KCSAN")
Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 mm/vma.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/mm/vma.h b/mm/vma.h
index 3fc3b47a95..4562bd4881 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -278,7 +278,13 @@ static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg)
  */
 static inline bool vma_has_anon_rmap(const struct vm_area_struct *vma)
 {
-	if (vma_is_attached(vma))
+	/*
+	 * Only check the locking under CONFIG_DEBUG_VM. vma_is_attached() and
+	 * vma_assert_locked() read vm_refcnt, which concurrent faults on the
+	 * same VMA keep writing, and __vmf_anon_prepare() calls this on the
+	 * anonymous page allocation and CoW fault paths.
+	 */
+	if (IS_ENABLED(CONFIG_DEBUG_VM) && vma_is_attached(vma))
 		vma_assert_stabilised(vma);
 	/* KCSAN gets confused about the optimistic check. Silence it. */
 	return data_race(vma->anon_vma);
-- 
2.53.0