[RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[tao]

Hi all,

Since v1 was relatively complex and difficult to extend, this version
adopts a simpler anon_vma implementation approach.


Replace the original anon_vma + anon_vma_chain multiple rbtree
structures with a depth-aware doubly linked list, and remove anon_vma_chain.


During rmap walks, vma_rmap_base(vma) is used to compute the page
address and look up the corresponding VMA from mm_mt:

  vma_rmap_base(vma) = vma->vm_start - vma->vm_pgoff * PAGE_SIZE

  page_address(vma, pgoff)
    = vma->vm_start + (pgoff - vma->vm_pgoff) * PAGE_SIZE
    = vma->vm_start - vma->vm_pgoff * PAGE_SIZE + pgoff * PAGE_SIZE
    = vma_rmap_base(vma) + pgoff * PAGE_SIZE

Since vma_rmap_base(vma) remains invariant across VMA merge and split
operations, VMA updates only need to adjust the VMA count of the
corresponding anon_vma. The anon_vma no longer directly tracks VMAs,
which eliminates the need for anon_vma_chain.

Process fork behaves like a fractal expansion of both tasks and
anon_vmas. A dedicated fractal_list records forked anon_vmas, while
descendant anon_vmas are derived from fork depth during rmap walks.
For this reason, the new design is named ANON_VMA_FRACTAL.

The depth is initialized to 0.
When a child is added, the root depth is set to 1.
A remap child has a depth of parent->depth + 1,
while a fork child has a depth of parent->depth + 2.
For example:
 1) A forks B and C:

	                   (1A)
	                  /    \
	                (3B)---(3C)

 2) B remaps D:

	                   (1A)
	                  /    \
	                (3B)   (3C)
	                /       /
	              (4D)-----o

 3) B forks E and F:

	                   (1A)--o
	                  /       \
	                (3B)      (3C)
	                /         /
	              (4D)       /
	              /         /
	            (5E)------(5F)

 4) C forks G:

	                   (1A)------------o
	                  /                 \
	                (3B)      (3C)       o
	                /         /  \      /
	              (4D)       /    \    /
	              /         /      \  /
	            (5E)------(5F)     (5G)

Only the root node A needs to traverse all descendants; non-root nodes
only traverse their own descendant nodes. For example:
- When rmap-ing A: first process A, then traverse all descendants
  B/D/E/F/C/G. A does not need to be traversed again.
- When rmap-ing B: first process B, then traverse its descendants
  D/E/F. Since C->depth <= B->depth, the rmap walk terminates at C.

The fractal_list remains stable across VMA merge and split operations.
It is updated only when anon_vmas are forked, remapped, or unlinked, and
is traversed during rmap walks.

On an Android device after boot, memory usage of vm_area_struct,
anon_vma, and anon_vma_chain is reduced by about 35 MB.

  struct              base(KB)  patch(KB)  patch_struct      saved(KB)
  --------------------------------------------------------------------
  vm_area_struct        110760     103837  vm_area_struct         6887
  anon_vma               19152       8576  anon_node             10576
  anon_vma_chain         17560          0  NA                    17560
  NA                         0        346  anon_semaphore         -346

lat_proc shows fork performance improves by about 15% at P=10.

Changes since v1:
- anon_vma now stores mm and rmap_base for VMA lookup, and only tracks
  the number of associated VMAs instead of directly tracking VMAs
- Use fractal_list + depth to maintain anon_vma topology and remove
  anon_vma_chain completely
- Keep folio->mapping storing anon_vma
- Preserve existing APIs and external callers unchanged while
  introducing ANON_RMAP_FOREACH_VMA() to unify anonymous rmap traversal
- Use vm_refcnt on leaf VMAs for rmap protection
- Optimize rwsem memory usage with shared semaphores

Patch layout:
  1-2: add CONFIG_ANON_VMA_FRACTAL and basic helpers
  3-7: implement anon_vma fractal infrastructure
  8: implement anonymous folio rmap
  9-10: replace anon_vma with anon_node
  11: use vm_refcnt on leaf VMAs for rmap protection
  12: optimize anon_vma memory usage with shared semaphores

v1:
https://lore.kernel.org/all/20260527110147.17815-1-tao.wangtao@honor.com/

tao (13):
  mm: add CONFIG_ANON_VMA_FRACTAL
  mm: implement helpers for ANON_VMA_FRACTAL
  mm: implement __anon_node_prepare for ANON_VMA_FRACTAL
  mm: implement anon_node_clone for ANON_VMA_FRACTAL
  mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL
  mm: implement unlink_anon_nodes for ANON_VMA_FRACTAL
  mm: handle rmap_base changes for ANON_VMA_FRACTAL
  mm: implement anonymous folio rmap for ANON_VMA_FRACTAL
  mm: prepare anon_node replacement for ANON_VMA_FRACTAL
  mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL
  mm: optimize rmap for ANON_VMA_FRACTAL with PVL
  mm: shared semaphores for ANON_VMA_FRACTAL
  mm: Enable CONFIG_ANON_VMA_FRACTAL by default

 include/linux/mm.h       |   2 +
 include/linux/mm_types.h |   2 +
 include/linux/rmap.h     |  56 ++++
 include/linux/rwsem.h    |  10 +
 mm/Kconfig               |  37 +++
 mm/internal.h            | 205 ++++++++++++
 mm/ksm.c                 |  19 +-
 mm/memory-failure.c      |   8 +-
 mm/mmap.c                |   7 +-
 mm/rmap.c                | 686 +++++++++++++++++++++++++++++++++++++--
 mm/vma.c                 |  76 ++++-
 mm/vma_init.c            |   2 +
 12 files changed, 1056 insertions(+), 54 deletions(-)

-- 
2.17.1

[RFC PATCH v2 01/13] mm: add CONFIG_ANON_VMA_FRACTAL

[tao]

Use vma_rmap_base(vma) to compute page addresses during rmap and
look up the corresponding VMA in mm_mt. Since vma_rmap_base(vma)
remains invariant across VMA merge and split, VMA operations only
need to adjust the VMA count of the corresponding anon_vma,
eliminating anon_vma_chain.

Fork creates derived anon_vmas in a fractal-like structure. A list
tracks anon_vmas, while fork depth is used to locate descendant
anon_vmas during rmap traversal.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 include/linux/rmap.h | 48 ++++++++++++++++++++++++++++++++++++++++++++
 mm/Kconfig           | 28 ++++++++++++++++++++++++++
 mm/internal.h        | 28 ++++++++++++++++++++++++++
 3 files changed, 104 insertions(+)

diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 8dc0871e5f00..75738f4ede1f 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -67,6 +67,54 @@ struct anon_vma {
 	struct rb_root_cached rb_root;
 };
 
+#ifdef CONFIG_ANON_VMA_FRACTAL
+
+/* struct anon_node - Replace anon_vma for anonymous page reverse mapping. */
+struct anon_node {
+	struct anon_node *root;		/* Root of this anon_node fractal tree */
+	struct rw_semaphore rwsem;
+	atomic_t vm_lock_ref;		/* for vm_lock_anon_vma */
+	atomic_t refcount;
+
+	struct anon_node *parent;	/* NULL if created by a page fault */
+
+	/*
+	 * depth is initialized to 0. When a child is added, depth is set to 1.
+	 * An rbc child has depth = parent->depth + 1,
+	 * and fork child has depth = parent->depth + 2.
+	 *
+	 * A forks B and C; B remaps D and forks E and F; C forks G.
+	 * The fractal_list is as follows:
+	 *                  (1A)--------------o
+	 *                 /                   \
+	 *               (3B)      (3C)         o
+	 *               /         /  \        /
+	 *             (4D)       /    \      /
+	 *             /         /      \    /
+	 *           (5E)----(5F)        (5G)
+	 */
+	unsigned int depth;
+	unsigned int nr_children;	/* < PID_MAX_LIMIT + max_map_count. */
+	struct list_head fractal_list;
+
+	/* mm and rmap_base can be used to find the VMA a page belongs to. */
+	struct mm_struct *mm;
+	/* rmap_base with VMA count in the low PAGE_SHIFT bits. */
+	atomic_long_t rbc;
+};
+
+/* Create rbc child anon_node on VMA remap or when count exceeds MAX. */
+#define ANON_RMAP_BASE_COUNT_MASK	(PAGE_SIZE - 1)
+#define ANON_RMAP_BASE_COUNT_MAX	(ANON_RMAP_BASE_COUNT_MASK)
+
+/*
+ * Allow VMAs to share an anon_node only if the number of VMAs attached
+ * is below this limit; keep it small to reduce lock contention.
+ */
+#define ANON_NODE_SHARE_LIMIT	(ANON_RMAP_BASE_COUNT_MAX / 1000)
+
+#endif
+
 /*
  * The copy-on-write semantics of fork mean that an anon_vma
  * can become associated with multiple processes. Furthermore,
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca4824905..c4c6d3e52d72 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1451,6 +1451,34 @@ config LOCK_MM_AND_FIND_VMA
 	bool
 	depends on !STACK_GROWSUP
 
+config ANON_VMA_FRACTAL
+	bool "anon_vma fractal list"
+	def_bool n
+	depends on MMU
+	help
+	  This option uses vma_rmap_base(vma) during rmap to compute the page
+	  address and look up the corresponding VMA in mm_mt.
+
+	    vma_rmap_base(vma) = vma->vm_start - vma->vm_pgoff * PAGE_SIZE
+
+	    page_address(vma, pgoff)
+	     = vma->vm_start + (pgoff - vma->vm_pgoff) * PAGE_SIZE
+	     = vma->vm_start - vma->vm_pgoff * PAGE_SIZE + pgoff * PAGE_SIZE
+	     = vma_rmap_base(vma) + pgoff * PAGE_SIZE
+
+	  Since vma_rmap_base(vma) remains invariant across VMA merge and
+	  split, VMA operations only need to adjust the VMA count of the
+	  corresponding anon_vma, eliminating the need for anon_vma_chain.
+
+	  Process fork behaves like a fractal expansion of both tasks and
+	  anon_vmas. A list records forked anon_vmas, and descendant anon_vmas
+	  are derived from fork depth for rmap handling.
+
+	  The fractal list remains unchanged during VMA merge or split. It
+	  is only updated when anon_vmas are created or removed, and traversed
+	  during rmap walks. Since rmap_one operations may switch tasks,
+	  semaphores are still required instead of spinlock_t protection.
+
 config IOMMU_MM_DATA
 	bool
 
diff --git a/mm/internal.h b/mm/internal.h
index 181e79f1d6a2..a33ed1a30f09 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -202,6 +202,16 @@ void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);
 
 #ifdef CONFIG_MMU
 
+static inline unsigned long rmap_base(unsigned long vm_start, pgoff_t pgoff)
+{
+	return vm_start - (pgoff << PAGE_SHIFT);
+}
+
+static inline unsigned long vma_rmap_base(const struct vm_area_struct *vma)
+{
+	return rmap_base(vma->vm_start, vma->vm_pgoff);
+}
+
 static inline void get_anon_vma(struct anon_vma *anon_vma)
 {
 	atomic_inc(&anon_vma->refcount);
@@ -245,6 +255,24 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
 	up_read(&anon_vma->root->rwsem);
 }
 
+#ifdef CONFIG_ANON_VMA_FRACTAL
+
+static inline struct anon_node *anon_node_next_rbc_child(
+		struct anon_node *anon_nod, struct anon_node *node)
+{
+	node = list_next_entry(node, fractal_list);
+	return node->depth == anon_nod->depth + 1 ? node : NULL;
+}
+
+static inline struct anon_node *anon_node_next_descendant(
+		struct anon_node *anon_nod, struct anon_node *node)
+{
+	node = list_next_entry(node, fractal_list);
+	return node->depth > anon_nod->depth ? node : NULL;
+}
+
+#endif
+
 struct anon_vma *folio_get_anon_vma(const struct folio *folio);
 
 /* Operations which modify VMAs. */
-- 
2.17.1

Re: [RFC PATCH v2 01/13] mm: add CONFIG_ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Pre-order tree traversal iterators are broken due to heterogeneous depth increments, causing cross-process memory corruption and missed VMAs during reverse mapping.
- [Medium] PAGE_SIZE-dependent VMA sharing limit causes severe memory bloat and arbitrary lock contention scaling.
- [Low] Duplicate and conflicting type directives for ANON_VMA_FRACTAL.
--

--- Patch [1]: [RFC PATCH v2 01/13] mm: add CONFIG_ANON_VMA_FRACTAL ---
commit 7f3f3b3fc54ea6c9180e3a0f624b7af6752d7ca5
Author: tao <tao.wangtao@honor.com>

mm: add CONFIG_ANON_VMA_FRACTAL

This commit adds the CONFIG_ANON_VMA_FRACTAL option to replace the
anon_vma_chain. It uses vma_rmap_base to compute page addresses and
organizes anon_vmas into a fractal list based on fork depth.

> diff --git a/include/linux/rmap.h b/include/linux/rmap.h
> index 8dc0871e5f001..75738f4ede1f6 100644
> --- a/include/linux/rmap.h
> +++ b/include/linux/rmap.h
[ ... ]
> @@ -67,6 +67,54 @@ struct anon_vma {
[ ... ]
> +/* Create rbc child anon_node on VMA remap or when count exceeds MAX. */
> +#define ANON_RMAP_BASE_COUNT_MASK	(PAGE_SIZE - 1)
> +#define ANON_RMAP_BASE_COUNT_MAX	(ANON_RMAP_BASE_COUNT_MASK)
> +
> +/*
> + * Allow VMAs to share an anon_node only if the number of VMAs attached
> + * is below this limit; keep it small to reduce lock contention.
> + */
> +#define ANON_NODE_SHARE_LIMIT	(ANON_RMAP_BASE_COUNT_MAX / 1000)

[Severity: Medium]
Since ANON_RMAP_BASE_COUNT_MAX relies on PAGE_SIZE, does this create an
artificially low sharing limit on 4K systems?

On a system with 4K pages, ANON_RMAP_BASE_COUNT_MAX would evaluate to 4095.
Dividing by 1000 means ANON_NODE_SHARE_LIMIT becomes 4, which would force a
new anon_node allocation very frequently and could cause memory bloat.

> +
> +#endif
> +
>  /*
>   * The copy-on-write semantics of fork mean that an anon_vma
>   * can become associated with multiple processes. Furthermore,
> diff --git a/mm/Kconfig b/mm/Kconfig
> index f794a341d7422..e0a1e96c6835a 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -1451,6 +1451,34 @@ config LOCK_MM_AND_FIND_VMA
>  	bool
>  	depends on !STACK_GROWSUP
>  
> +config ANON_VMA_FRACTAL
> +	bool "anon_vma fractal list"
> +	def_bool n

[Severity: Low]
This isn't a bug, but using both a bool prompt and def_bool together
conflicts with standard Kconfig syntax. Could we just use bool and
default n instead?

> +	depends on MMU
> +	help
> +	  This option uses vma_rmap_base(vma) during rmap to compute the page
[ ... ]
> diff --git a/mm/internal.h b/mm/internal.h
> index 841c27611627e..694d3194a6baf 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
[ ... ]
> @@ -252,6 +262,24 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
>  	up_read(&anon_vma->root->rwsem);
>  }
>  
> +#ifdef CONFIG_ANON_VMA_FRACTAL
> +
> +static inline struct anon_node *anon_node_next_rbc_child(
> +		struct anon_node *anon_nod, struct anon_node *node)
> +{
> +	node = list_next_entry(node, fractal_list);
> +	return node->depth == anon_nod->depth + 1 ? node : NULL;
> +}

[Severity: Critical]
Will this iteration abort prematurely if the next entry in the list is a
descendant of the RBC child?

If an RBC child has its own descendants, they would immediately follow it in
the list. Since anon_node_next_rbc_child() in mm/internal.h strictly expects
the very next list entry to have depth == anon_nod->depth + 1, it appears it
would return NULL and stop iteration early, potentially missing VMAs.

> +
> +static inline struct anon_node *anon_node_next_descendant(
> +		struct anon_node *anon_nod, struct anon_node *node)
> +{
> +	node = list_next_entry(node, fractal_list);
> +	return node->depth > anon_nod->depth ? node : NULL;
> +}

[Severity: Critical]
Could this traversal yield unrelated siblings as descendants when depth
increments are mixed?

Since RBC children increment depth by 1 and fork children increment by 2, a
process calling mremap (creating an RBC child) followed by fork (creating a
fork child) will create mixed increments.

If we iterate descendants of an RBC child (depth P+1),
anon_node_next_descendant() in mm/internal.h would continue as long as
node->depth > anon_nod->depth. A sibling fork child with depth P+2 would
satisfy this condition and incorrectly be yielded as a descendant,
potentially leading to cross-process memory corruption during rmap
traversals.

> +
> +#endif
> +
>  struct anon_vma *folio_get_anon_vma(const struct folio *folio);
>  
>  /* Operations which modify VMAs. */

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=1

[RFC PATCH v2 02/13] mm: implement helpers for ANON_VMA_FRACTAL

[tao]

Implement basic helpers.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 25 +++++++++++++++
 mm/rmap.c     | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)

diff --git a/mm/internal.h b/mm/internal.h
index a33ed1a30f09..c1b3961b9d3e 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -271,6 +271,31 @@ static inline struct anon_node *anon_node_next_descendant(
 	return node->depth > anon_nod->depth ? node : NULL;
 }
 
+static inline void get_anon_node(struct anon_node *anon_nod)
+{
+	VM_BUG_ON(atomic_read(&anon_nod->refcount) == 0);
+	atomic_inc(&anon_nod->refcount);
+}
+
+void __put_anon_node(struct anon_node *anon_nod);
+
+static inline void put_anon_node(struct anon_node *anon_nod)
+{
+	VM_BUG_ON(atomic_read(&anon_nod->refcount) == 0);
+	if (atomic_dec_and_test(&anon_nod->refcount))
+		__put_anon_node(anon_nod);
+}
+
+static inline unsigned long anon_node_rmap_base(struct anon_node *anon_nod)
+{
+	return atomic_long_read(&anon_nod->rbc) & ~ANON_RMAP_BASE_COUNT_MASK;
+}
+
+static inline unsigned long anon_node_rmap_count(struct anon_node *anon_nod)
+{
+	return atomic_long_read(&anon_nod->rbc) & ANON_RMAP_BASE_COUNT_MAX;
+}
+
 #endif
 
 struct anon_vma *folio_get_anon_vma(const struct folio *folio);
diff --git a/mm/rmap.c b/mm/rmap.c
index 1c77d5dc06e9..fa9d82271005 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -558,6 +558,95 @@ void __init anon_vma_init(void)
 			SLAB_PANIC|SLAB_ACCOUNT);
 }
 
+#ifdef CONFIG_ANON_VMA_FRACTAL
+
+static struct kmem_cache *anon_node_cachep;
+
+static inline struct rw_semaphore *anon_node_rmap_sem(struct anon_node *node)
+{
+	return &node->root->rwsem;
+}
+
+static inline atomic_t *anon_node_vm_lock_ref(struct anon_node *node)
+{
+	return &node->root->vm_lock_ref;
+}
+
+static void anon_node_ctor(void *data)
+{
+	struct anon_node *anon_nod = data;
+
+	init_rwsem(&anon_nod->rwsem);
+	atomic_set(&anon_nod->vm_lock_ref, 0);
+	atomic_set(&anon_nod->refcount, 0);
+}
+
+static inline void anon_vma_fractal_init(void)
+{
+	struct kmem_cache_args args = {
+		.use_freeptr_offset = true,
+		.freeptr_offset = offsetof(struct anon_node, fractal_list),
+		.ctor = anon_node_ctor,
+	};
+
+	anon_node_cachep = kmem_cache_create("anon_node",
+			sizeof(struct anon_node), &args,
+			SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT);
+}
+
+static inline void anon_node_free(struct anon_node *anon_nod)
+{
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
+
+	VM_BUG_ON(atomic_read(&anon_nod->refcount));
+	VM_BUG_ON(!list_empty(&anon_nod->fractal_list));
+	VM_BUG_ON(anon_nod->nr_children);
+	VM_BUG_ON(anon_node_rmap_count(anon_nod));
+
+	might_sleep();
+	if (rwsem_is_locked(rmap_sem)) {
+		down_write(rmap_sem);
+		up_write(rmap_sem);
+	}
+
+	kmem_cache_free(anon_node_cachep, anon_nod);
+}
+
+void __put_anon_node(struct anon_node *anon_nod)
+{
+	struct anon_node *root = anon_nod->root;
+
+	anon_node_free(anon_nod);
+	if (root != anon_nod && atomic_dec_and_test(&root->refcount))
+		anon_node_free(root);
+}
+
+static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
+		struct anon_node *parent, bool is_fork)
+{
+	struct anon_node *anon_nod;
+
+	anon_nod = kmem_cache_alloc(anon_node_cachep, GFP_KERNEL);
+	if (!anon_nod)
+		return NULL;
+
+	anon_nod->root = parent ? parent->root : anon_nod;
+	anon_nod->parent = parent;
+	atomic_set(&anon_nod->refcount, 1);
+	anon_nod->depth = 0;
+	if (parent) {
+		get_anon_node(parent->root);
+		anon_nod->depth = parent->depth + (1 << is_fork);
+	}
+	anon_nod->nr_children = 0;
+	INIT_LIST_HEAD(&anon_nod->fractal_list);
+	anon_nod->mm = vma->vm_mm;
+	atomic_long_set(&anon_nod->rbc, vma_rmap_base(vma) + 1);
+	return anon_nod;
+}
+
+#endif
+
 /*
  * Getting a lock on a stable anon_vma from a page off the LRU is tricky!
  *
-- 
2.17.1

Re: [RFC PATCH v2 02/13] mm: implement helpers for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] `anon_node_alloc()` fails to take a reference on the intermediate `parent` node, leading to premature freeing and dangling parent pointers.
- [High] `anon_vma_fractal_init()` is defined but never called, leaving `anon_node_cachep` uninitialized.
- [Medium] The `anon_node` structure caches a raw pointer to an `mm_struct` without holding a reference, which can lead to a Use-After-Free.
- [Low] Unconditional access to conditionally compiled fields `rwsem` and `vm_lock_ref` breaks the build when `CONFIG_ANON_VMA_SHARED_SEMS` is enabled.
--

--- Patch [2]: [RFC PATCH v2 02/13] mm: implement helpers for ANON_VMA_FRACTAL ---
commit 353e80074df4a13790b7dd27f2e9eb1a62294b38
Author: tao <tao.wangtao@honor.com>

mm: implement helpers for ANON_VMA_FRACTAL
    
Implement basic helpers.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index 9939400e77c79..54c5ccfee6813 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
[ ... ]
> @@ -558,6 +558,95 @@ void __init anon_vma_init(void)
>  			SLAB_PANIC|SLAB_ACCOUNT);
>  }
>  
> +#ifdef CONFIG_ANON_VMA_FRACTAL
> +
> +static struct kmem_cache *anon_node_cachep;
> +
> +static inline struct rw_semaphore *anon_node_rmap_sem(struct anon_node *node)
> +{
> +	return &node->root->rwsem;
> +}
> +
> +static inline atomic_t *anon_node_vm_lock_ref(struct anon_node *node)
> +{
> +	return &node->root->vm_lock_ref;
> +}
> +
> +static void anon_node_ctor(void *data)
> +{
> +	struct anon_node *anon_nod = data;
> +
> +	init_rwsem(&anon_nod->rwsem);
> +	atomic_set(&anon_nod->vm_lock_ref, 0);
> +	atomic_set(&anon_nod->refcount, 0);
> +}

[Severity: Low]
Does this cause a build failure when CONFIG_ANON_VMA_SHARED_SEMS is enabled?

The rwsem and vm_lock_ref fields in struct anon_node are conditionally
compiled out when this config is set, but they are accessed unconditionally
here in anon_node_ctor() and anon_node_rmap_sem().

> +
> +static inline void anon_vma_fractal_init(void)
> +{
> +	struct kmem_cache_args args = {
> +		.use_freeptr_offset = true,
> +		.freeptr_offset = offsetof(struct anon_node, fractal_list),
> +		.ctor = anon_node_ctor,
> +	};
> +
> +	anon_node_cachep = kmem_cache_create("anon_node",
> +			sizeof(struct anon_node), &args,
> +			SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT);
> +}

[Severity: High]
Is anon_vma_fractal_init() ever called?

It doesn't appear to be hooked up to anon_vma_init() or anywhere else, which
would leave anon_node_cachep uninitialized and result in a null pointer
dereference crash on the first allocation.

[ ... ]
> +static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
> +		struct anon_node *parent, bool is_fork)
> +{
> +	struct anon_node *anon_nod;
> +
> +	anon_nod = kmem_cache_alloc(anon_node_cachep, GFP_KERNEL);
> +	if (!anon_nod)
> +		return NULL;
> +
> +	anon_nod->root = parent ? parent->root : anon_nod;
> +	anon_nod->parent = parent;
> +	atomic_set(&anon_nod->refcount, 1);
> +	anon_nod->depth = 0;
> +	if (parent) {
> +		get_anon_node(parent->root);
> +		anon_nod->depth = parent->depth + (1 << is_fork);
> +	}

[Severity: High]
Does this code leave the intermediate parent node vulnerable to being freed?

We assign anon_nod->parent = parent but only take a reference on parent->root.
If the intermediate parent is freed while this node is still alive,
anon_nod->parent becomes a dangling pointer.

> +	anon_nod->nr_children = 0;
> +	INIT_LIST_HEAD(&anon_nod->fractal_list);
> +	anon_nod->mm = vma->vm_mm;

[Severity: Medium]
Should we acquire a reference to the mm_struct using mmgrab() here?

A process exiting will free its mm_struct. If child processes keep the
parent's root anon_node alive via get_anon_node(parent->root), the
anon_node->mm pointer could become a dangling pointer, leading to a
use-after-free during lockless rmap walks.

> +	atomic_long_set(&anon_nod->rbc, vma_rmap_base(vma) + 1);
> +	return anon_nod;
> +}
> +
> +#endif

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=2

[RFC PATCH v2 03/13] mm: implement __anon_node_prepare for ANON_VMA_FRACTAL

[tao]

Add anon_node APIs corresponding to existing anon_vma operations:

- anon_node_clone() corresponds to anon_vma_clone()
- __anon_node_prepare() corresponds to __anon_vma_prepare()
- anon_node_fork_with_prev() corresponds to anon_vma_fork()
- unlink_anon_nodes() corresponds to unlink_anon_vmas()

Implement __anon_node_prepare() first to keep the patch small.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 11 ++++++++++
 mm/rmap.c     | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/mm/internal.h b/mm/internal.h
index c1b3961b9d3e..710baf02575b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -314,6 +314,17 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma);
 int  __anon_vma_prepare(struct vm_area_struct *vma);
 void unlink_anon_vmas(struct vm_area_struct *vma);
 
+#ifdef CONFIG_ANON_VMA_FRACTAL
+int anon_node_clone(struct vm_area_struct *vma, struct vm_area_struct *src,
+	enum vma_operation operation);
+int anon_node_fork_with_prev(struct vm_area_struct *vma,
+	struct vm_area_struct *pvma, struct vm_area_struct *vma_prev,
+	struct vm_area_struct *pvma_prev);
+int __anon_node_prepare(struct vm_area_struct *vma);
+void unlink_anon_nodes(struct vm_area_struct *vma);
+
+#endif
+
 static inline int anon_vma_prepare(struct vm_area_struct *vma)
 {
 	if (likely(vma->anon_vma))
diff --git a/mm/rmap.c b/mm/rmap.c
index fa9d82271005..b98ba6a83272 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -645,6 +645,64 @@ static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
 	return anon_nod;
 }
 
+static bool try_track_rmap(struct anon_node *anon_nod, unsigned long rmap_base)
+{
+	const unsigned long rbc_max = rmap_base + ANON_RMAP_BASE_COUNT_MAX;
+
+	return rmap_base == anon_node_rmap_base(anon_nod) &&
+		atomic_long_add_unless(&anon_nod->rbc, 1, rbc_max);
+}
+
+static inline bool try_untrack_rmap(struct anon_node *anon_nod,
+		unsigned long rmap_base, bool *test_empty)
+{
+	const unsigned long rbc_max = rmap_base + ANON_RMAP_BASE_COUNT_MAX;
+	unsigned long rbc = atomic_long_read(&anon_nod->rbc);
+
+	if (rbc > rmap_base && rbc <= rbc_max) {
+		rbc = atomic_long_dec_return(&anon_nod->rbc);
+		*test_empty = (rbc == rmap_base);
+		return true;
+	}
+	return false;
+}
+
+/* attach an anon_node to the VMA. */
+int __anon_node_prepare(struct vm_area_struct *vma)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	unsigned long rmap_base = vma_rmap_base(vma);
+	struct anon_node *anon_nod;
+	struct anon_vma *anon_old = NULL, *anon_new;
+	bool test_empty = false;
+
+	mmap_assert_locked(mm);
+	might_sleep();
+
+	anon_nod = NULL; /* Fix find_mergeable_anon_vma(vma) later. */
+	if (anon_nod) {
+		anon_new = (struct anon_vma *)anon_nod;
+		if (try_track_rmap(anon_nod, rmap_base) &&
+		    !try_cmpxchg(&vma->anon_vma, &anon_old, anon_new))
+			try_untrack_rmap(anon_nod, rmap_base, &test_empty);
+		if (vma->anon_vma)
+			return 0;
+	}
+
+	anon_nod = anon_node_alloc(vma, NULL, false);
+	if (unlikely(!anon_nod))
+		return -ENOMEM;
+
+	anon_new = (struct anon_vma *)anon_nod;
+	/* maybe prepared by another thread */
+	if (!try_cmpxchg(&vma->anon_vma, &anon_old, anon_new)) {
+		try_untrack_rmap(anon_nod, rmap_base, &test_empty);
+		put_anon_node(anon_nod);
+	}
+
+	return 0;
+}
+
 #endif
 
 /*
-- 
2.17.1

Re: [RFC PATCH v2 03/13] mm: implement __anon_node_prepare for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] `__anon_node_prepare` casts a `struct anon_node *` to a `struct anon_vma *` and assigns it to `vma->anon_vma`, resulting in severe type confusion.
--

--- Patch [3]: [RFC PATCH v2 03/13] mm: implement __anon_node_prepare for ANON_VMA_FRACTAL ---
commit 1ab3065664fb86573b65c2ad490a2b4e7c508707
Author: tao <tao.wangtao@honor.com>

    mm: implement __anon_node_prepare for ANON_VMA_FRACTAL

This commit introduces anon_node APIs corresponding to existing anon_vma
operations for the ANON_VMA_FRACTAL feature. It currently implements
__anon_node_prepare() to attach an anon_node to a VMA.

> diff --git a/mm/rmap.c b/mm/rmap.c
> --- a/mm/rmap.c
> +++ b/mm/rmap.c

[ ... ]

> +int __anon_node_prepare(struct vm_area_struct *vma)
> +{
> +	struct mm_struct *mm = vma->vm_mm;
> +	unsigned long rmap_base = vma_rmap_base(vma);
> +	struct anon_node *anon_nod;
> +	struct anon_vma *anon_old = NULL, *anon_new;

[ ... ]

> +	anon_nod = anon_node_alloc(vma, NULL, false);
> +	if (unlikely(!anon_nod))
> +		return -ENOMEM;
> +
> +	anon_new = (struct anon_vma *)anon_nod;
> +	/* maybe prepared by another thread */
> +	if (!try_cmpxchg(&vma->anon_vma, &anon_old, anon_new)) {

[Severity: Critical]
Will this cause memory corruption for existing reverse mapping and page fault
paths that expect a standard anon_vma?

Because struct anon_node and struct anon_vma have significantly different
memory layouts, storing an anon_node pointer into vma->anon_vma will cause
type confusion. For instance, anon_node places vm_lock_ref at the exact
memory offset where anon_vma expects refcount. 

When standard VMA operations (like folio_lock_anon_vma_read() or
unlink_anon_vmas()) access vma->anon_vma->refcount, won't they read and write
the wrong fields, corrupting vm_lock_ref and permanently leaking the anon_node
since its actual refcount is never decremented?

> +		try_untrack_rmap(anon_nod, rmap_base, &test_empty);
> +		put_anon_node(anon_nod);
> +	}
> +
> +	return 0;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=3

[RFC PATCH v2 04/13] mm: implement anon_node_clone for ANON_VMA_FRACTAL

[tao]

Prefer using anon_node for tracking in anon_node_clone.
If the VMA is remapped or the VMA count exceeds
ANON_RMAP_BASE_COUNT_MAX, create a child anon_node.

Newly created anon_node are inserted at the tail of the rbc children.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/rmap.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/mm/rmap.c b/mm/rmap.c
index b98ba6a83272..74817876107a 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -703,6 +703,91 @@ int __anon_node_prepare(struct vm_area_struct *vma)
 	return 0;
 }
 
+static inline struct anon_node *vma_anon_node(struct vm_area_struct *vma)
+{
+	return vma ? (struct anon_node *)vma->anon_vma : NULL;
+}
+
+static inline void vma_set_anon_node(struct vm_area_struct *vma,
+		struct anon_node *anon_nod, bool is_locked)
+{
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
+
+	if (is_locked) {
+		vma->anon_vma = (struct anon_vma *)anon_nod;
+		return;
+	}
+	down_write(rmap_sem);
+	vma->anon_vma = (struct anon_vma *)anon_nod;
+	up_write(rmap_sem);
+}
+
+static int anon_node_add_child(struct anon_node *anon_nod,
+	struct vm_area_struct *vma, bool is_fork)
+{
+	struct anon_node *child;
+	struct anon_node *root = anon_nod->root;
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(root);
+	struct anon_node *rbc_ch = anon_nod, *next;
+
+	if (root->depth == 0)
+		root->depth = 1;
+	child = anon_node_alloc(vma, anon_nod, is_fork);
+	if (!child)
+		return -ENOMEM;
+
+	rmap_sem = anon_node_rmap_sem(root);
+	down_write(rmap_sem);
+	/* Add new child to the tail of rbc children. */
+	while ((next = anon_node_next_rbc_child(anon_nod, rbc_ch)) != NULL) {
+		rbc_ch = next;
+	}
+	list_add(&child->fractal_list, &rbc_ch->fractal_list);
+	++anon_nod->nr_children;
+
+	if (is_fork)
+		vma_set_anon_node(vma, child, true);
+	up_write(rmap_sem);
+	return 0;
+}
+
+/* Returns 0 on success, non-zero on failure. */
+static int anon_node_track_rmap(struct anon_node *anon_nod,
+		struct vm_area_struct *vma)
+{
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
+	unsigned long rmap_base = vma_rmap_base(vma);
+	struct anon_node *rbc_ch = anon_nod;
+
+	if (try_track_rmap(anon_nod, rmap_base))
+		return 0;
+
+	down_read(rmap_sem);
+	while ((rbc_ch = anon_node_next_rbc_child(anon_nod, rbc_ch)) != NULL) {
+		if (try_track_rmap(rbc_ch, rmap_base))
+			break;
+	}
+	up_read(rmap_sem);
+	if (rbc_ch)
+		return 0;
+
+	return anon_node_add_child(anon_nod, vma, false);
+}
+
+/* vma clones src anon_node and increments the count. */
+int anon_node_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
+	enum vma_operation operation)
+{
+	struct anon_node *anon_nod = vma_anon_node(src);
+
+	mmap_assert_write_locked(vma->vm_mm);
+	if (!anon_nod)
+		return 0;
+
+	VM_BUG_ON_VMA(vma->anon_vma != (void *)anon_nod, vma);
+	return anon_node_track_rmap(anon_nod, vma);
+}
+
 #endif
 
 /*
-- 
2.17.1

Re: [RFC PATCH v2 04/13] mm: implement anon_node_clone for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] Flattening the fractal tree into a single list corrupts child iteration when children have their own descendants, permanently severing the tree structure.
- [Critical] VMA unlinking unconditionally casts anon_node to anon_vma and overwrites mismatched fields.
- [High] Unsynchronized read-modify-write of root->depth causes data races and structural corruption of the fractal tree.
- [High] Newly allocated anon_node objects are permanently leaked because their reference counts are never decremented during VMA teardown.
- [High] anon_node objects are never removed from fractal_list, leading to a Kernel BUG and Use-After-Free.
- [High] Intermediate anon_node structures can be freed while they still have children in their fractal_list, leading to a VM_BUG_ON panic or Use-After-Free.
- [High] anon_node slab objects allocated in anon_node_track_rmap() with is_fork=false are permanently leaked because ownership is never transferred to a VMA.
- [Medium] Missing state re-validation after lock upgrade in anon_node_track_rmap() leads to duplicate child node insertions.
--

--- Patch [4]: [RFC PATCH v2 04/13] mm: implement anon_node_clone for ANON_VMA_FRACTAL ---
commit d6053627649639aa9d6f587172befdbc0e783aed
Author: tao <tao.wangtao@honor.com>

mm: implement anon_node_clone for ANON_VMA_FRACTAL

This commit implements anon_node_clone for ANON_VMA_FRACTAL. It creates
a child anon_node when the VMA is remapped or when the VMA count exceeds
the maximum trackable limit. Newly created nodes are inserted at the tail
of the rbc children list.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index 55074ee931967..9dc82e7438b00 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c

[ ... ]

> +static inline void vma_set_anon_node(struct vm_area_struct *vma,
> +		struct anon_node *anon_nod, bool is_locked)
> +{
> +	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
> +
> +	if (is_locked) {
> +		vma->anon_vma = (struct anon_vma *)anon_nod;
> +		return;
> +	}
> +	down_write(rmap_sem);
> +	vma->anon_vma = (struct anon_vma *)anon_nod;

[Severity: Critical]
Does this type cast corrupt the anon_node structure during teardown?

During teardown, the standard unlink_anon_vmas() flow operates on this pointer
and executes active_anon_vma->num_active_vmas--. Since anon_node has a
different layout and lacks num_active_vmas, won't this unconditionally write
out of bounds and corrupt the anon_node fields?

> +	up_write(rmap_sem);
> +}
> +
> +static int anon_node_add_child(struct anon_node *anon_nod,
> +	struct vm_area_struct *vma, bool is_fork)
> +{
> +	struct anon_node *child;
> +	struct anon_node *root = anon_nod->root;
> +	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(root);
> +	struct anon_node *rbc_ch = anon_nod, *next;
> +
> +	if (root->depth == 0)
> +		root->depth = 1;

[Severity: High]
Can this unsynchronized read-modify-write cause data races?

root->depth is being modified outside of any locks. Could a concurrent thread
allocating a node via anon_node_alloc() read an incorrect parent->depth of 0
instead of 1? Because tree traversals require accurate depths, won't an
incorrectly initialized branch permanently break the tree structure?

> +	child = anon_node_alloc(vma, anon_nod, is_fork);

[Severity: High]
Are these newly allocated anon_node objects permanently leaked?

anon_node is allocated with a refcount of 1. During VMA teardown,
unlink_anon_vmas() relies on iterating vma->anon_vma_chain to call
put_anon_vma(). However, since anon_node logic never populates anon_vma_chain,
won't the iteration loop silently skip over it and leak the memory?

[Severity: High]
Does anon_node_alloc() fail to pin the direct parent node?

In anon_node_alloc(), the logic increments the refcount of the root node but
doesn't increment the refcount of the direct parent. If an intermediate parent
node is unmapped, won't its refcount drop to 0 and get freed while it still
has children in its fractal_list? This could lead to a VM_BUG_ON panic or a
use-after-free for the children.

> +	if (!child)
> +		return -ENOMEM;
> +
> +	rmap_sem = anon_node_rmap_sem(root);
> +	down_write(rmap_sem);
> +	/* Add new child to the tail of rbc children. */
> +	while ((next = anon_node_next_rbc_child(anon_nod, rbc_ch)) != NULL) {
> +		rbc_ch = next;
> +	}
> +	list_add(&child->fractal_list, &rbc_ch->fractal_list);

[Severity: Critical]
Does inserting the new child here corrupt the contiguous list iteration?

The iterator macro anon_node_next_rbc_child() strictly expects contiguous
children and returns NULL when node->depth != parent->depth + 1:

mm/internal.h:anon_node_next_rbc_child() {
    ...
    return node->depth == anon_nod->depth + 1 ? node : NULL;
}

If an existing child has its own descendants, those descendants interrupt the
contiguous sequence. The loop above stops prematurely at the first child with
descendants. By inserting the new child via list_add() immediately after that
child, won't the new sibling be placed before the descendants? This would
break the depth check for both anon_node_next_rbc_child() and
anon_node_next_descendant(), permanently severing all subsequent subtrees.

[Severity: High]
Are anon_node objects ever removed from fractal_list during teardown?

Nodes are added to the list here, but it doesn't appear there is a
corresponding list_del() anywhere in the teardown logic. If anon_node_free()
were reached, wouldn't it assert VM_BUG_ON(!list_empty(&anon_nod->fractal_list))?
Furthermore, freeing a node without removing it from the list would leave
dangling list pointers in its parent and siblings, causing a use-after-free
on the next tree traversal.

> +	++anon_nod->nr_children;
> +
> +	if (is_fork)
> +		vma_set_anon_node(vma, child, true);

[Severity: High]
Are anon_node objects allocated with is_fork=false permanently leaked?

When anon_node_track_rmap() expands tracking by calling anon_node_add_child()
with is_fork=false, the vma_set_anon_node() call here is skipped. Because the
newly allocated child node is never attached to a VMA, is there any mechanism
to track or decrement its initial reference count when the VMA is ultimately
unmapped?

> +	up_write(rmap_sem);
> +	return 0;
> +}
> +
> +/* Returns 0 on success, non-zero on failure. */
> +static int anon_node_track_rmap(struct anon_node *anon_nod,
> +		struct vm_area_struct *vma)
> +{
> +	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
> +	unsigned long rmap_base = vma_rmap_base(vma);
> +	struct anon_node *rbc_ch = anon_nod;
> +
> +	if (try_track_rmap(anon_nod, rmap_base))
> +		return 0;
> +
> +	down_read(rmap_sem);
> +	while ((rbc_ch = anon_node_next_rbc_child(anon_nod, rbc_ch)) != NULL) {
> +		if (try_track_rmap(rbc_ch, rmap_base))
> +			break;
> +	}
> +	up_read(rmap_sem);
> +	if (rbc_ch)
> +		return 0;
> +
> +	return anon_node_add_child(anon_nod, vma, false);

[Severity: Medium]
Should the state be re-validated after acquiring the write lock?

anon_node_track_rmap() searches for a capable child node under a read lock.
If it fails, it drops the read lock and calls anon_node_add_child(), which
takes a write lock to unconditionally allocate and append a node.

Could concurrent threads that fail the read check queue up on the write lock
and blindly append redundant tracking nodes for the exact same rmap_base?

> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=4

[RFC PATCH v2 05/13] mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL

[tao]

Fork handling introduces two additional parameters to help child VMAs
clone anon_node. The logic follows three cases:
1) Fast path: clone the anon_node with the previous VMA.
2) Reuse an empty anon_node from an ancestor.
3) Allocate a new anon_node and insert it into fractal_list.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/rmap.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

diff --git a/mm/rmap.c b/mm/rmap.c
index 74817876107a..93fa04c5a007 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -667,6 +667,17 @@ static inline bool try_untrack_rmap(struct anon_node *anon_nod,
 	return false;
 }
 
+static bool try_reuse_anon_node(struct anon_node *anon_nod,
+		unsigned long new_rbc)
+{
+	long old_rbase;
+
+	if (anon_node_rmap_count(anon_nod))
+		return false;
+	old_rbase = anon_node_rmap_base(anon_nod);
+	return atomic_long_try_cmpxchg(&anon_nod->rbc, &old_rbase, new_rbc);
+}
+
 /* attach an anon_node to the VMA. */
 int __anon_node_prepare(struct vm_area_struct *vma)
 {
@@ -788,6 +799,67 @@ int anon_node_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
 	return anon_node_track_rmap(anon_nod, vma);
 }
 
+static struct anon_node *fork_reuse_ancestor(struct anon_node *parent,
+		struct vm_area_struct *vma)
+{
+	struct anon_node *root = parent->root;
+	struct anon_node *node;
+	unsigned long new_rbc = vma_rmap_base(vma) + 1;
+	/* At least N anon_nodes between parent and root: P -> a1 .. aN -> R */
+	const unsigned int ANON_DEPTH_REUSE_THRESHOLD = 1 + (5 << 1);
+
+	if (parent->depth < ANON_DEPTH_REUSE_THRESHOLD)
+		return NULL;
+
+	/* Release from leaf only; parent is stable, no lock needed. */
+	for (node = parent; node != root; node = node->parent) {
+		if (READ_ONCE(node->nr_children) > 1)
+			continue;
+		/* Replace with the new_rbc atomically. */
+		if (!try_reuse_anon_node(node, new_rbc))
+			continue;
+
+		node->mm = vma->vm_mm;
+		vma_set_anon_node(vma, node, false);
+		return node;
+	}
+
+	return NULL;
+}
+
+/* Clone an anon_node from prev or fork one from the parent pvma. */
+int anon_node_fork_with_prev(struct vm_area_struct *vma,
+	struct vm_area_struct *pvma, struct vm_area_struct *prev,
+	struct vm_area_struct *pvma_prev)
+{
+	struct anon_node *parent = vma_anon_node(pvma);
+	struct anon_node *anon_nod = vma_anon_node(prev);
+	unsigned long rmap_base = vma_rmap_base(vma);
+
+	if (!parent)
+		return 0;
+
+	/* Drop inherited anon_nod. */
+	vma->anon_vma = NULL;
+
+	/* 1) Fast path: clone the anon_node with the previous VMA. */
+	if (anon_nod && pvma_prev && parent == vma_anon_node(pvma_prev)) {
+		if (try_track_rmap(anon_nod, rmap_base)) {
+			vma_set_anon_node(vma, anon_nod, false);
+			return 0;
+		}
+	}
+
+	/* 2) Reuse an existing anon_node. */
+	anon_nod = fork_reuse_ancestor(parent, vma);
+	if (anon_nod)
+		return 0;
+
+	/* 3) Fork a new anon_node. */
+	return anon_node_add_child(parent, vma, true);
+}
+
+
 #endif
 
 /*
-- 
2.17.1

Re: [RFC PATCH v2 05/13] mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Critical] Missing child-to-parent reference pinning allows parent `anon_node`s to be freed while descendants retain dangling pointers.
- [Critical] Reusing an ancestor `anon_node` for a fork child VMA breaks the reverse mapping hierarchy, preventing shared pages from being unmapped.
- [High] Lockless update of `node->mm` in `fork_reuse_ancestor()` creates a data race with concurrent page table walkers, leading to Use-After-Free.
- [High] Child `anon_node`s are never unlinked from `fractal_list`, causing `anon_node_free()` to trigger `VM_BUG_ON` and leaving dangling pointers.
- [High] Missing `unlink_anon_nodes` implementation results in a permanent memory leak of all `anon_node` structs.
--

--- Patch [5]: [RFC PATCH v2 05/13] mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL ---
commit 24cea8e5be0b7dbc9e5d9160c41d3819f2dc0c2e
Author: tao <tao.wangtao@honor.com>

mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL

Fork handling introduces two additional parameters to help child VMAs
clone anon_node. The logic follows three cases:
1) Fast path: clone the anon_node with the previous VMA.
2) Reuse an empty anon_node from an ancestor.
3) Allocate a new anon_node and insert it into fractal_list.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index 9dc82e7438b00..9e86f64ea9e53 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c

[ ... ]

> @@ -788,6 +799,67 @@ int anon_node_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
>  	return anon_node_track_rmap(anon_nod, vma);
>  }
>  
> +static struct anon_node *fork_reuse_ancestor(struct anon_node *parent,
> +		struct vm_area_struct *vma)
> +{
> +	struct anon_node *root = parent->root;
> +	struct anon_node *node;
> +	unsigned long new_rbc = vma_rmap_base(vma) + 1;
> +	/* At least N anon_nodes between parent and root: P -> a1 .. aN -> R */
> +	const unsigned int ANON_DEPTH_REUSE_THRESHOLD = 1 + (5 << 1);
> +
> +	if (parent->depth < ANON_DEPTH_REUSE_THRESHOLD)
> +		return NULL;
> +
> +	/* Release from leaf only; parent is stable, no lock needed. */
> +	for (node = parent; node != root; node = node->parent) {

[Severity: Critical]
Does this traverse upwards using the unpinned node->parent pointer?

It looks like anon_node_alloc() pins the root but fails to increment the
refcount of its immediate parent:

mm/rmap.c:anon_node_alloc() {
    ...
    anon_nod->parent = parent;
    anon_nod->depth = parent->depth + (is_fork ? 2 : 1);
    get_anon_node(parent->root);
    ...
}

If all VMAs attached to the parent are unmapped, the parent node's refcount
could drop to 0 and be freed. Could this regression result in a use-after-free
when this loop traverses node = node->parent?

> +		if (READ_ONCE(node->nr_children) > 1)
> +			continue;
> +		/* Replace with the new_rbc atomically. */
> +		if (!try_reuse_anon_node(node, new_rbc))
> +			continue;
> +
> +		node->mm = vma->vm_mm;

[Severity: High]
Can updating node->mm locklessly after try_reuse_anon_node() cause a
use-after-free regression?

try_reuse_anon_node() atomically updates rbc, which publishes the node as active
to concurrent rmap walkers traversing the fractal_list.

Since node->mm is updated after this atomic operation, could a concurrent
rmap walk observe the active node via rbc but read the old, potentially freed
node->mm pointer before the assignment completes?

> +		vma_set_anon_node(vma, node, false);
> +		return node;

[Severity: Critical]
Does reusing an ancestor anon_node for a fork child VMA break the reverse
mapping hierarchy?

The reverse mapping algorithm relies on traversing downwards from the anon_node
where a page was faulted to find all descendant VMAs sharing the page.

If an empty ancestor node is assigned to the fork child without relocating it
in the tree, shared pages mapped by the parent before the fork will point to
the parent's anon_node.

When rmap_walk starts at the parent, will it completely miss the child VMA
because its node (the ancestor) is structurally above the parent?

> +	}
> +
> +	return NULL;
> +}
> +
> +/* Clone an anon_node from prev or fork one from the parent pvma. */
> +int anon_node_fork_with_prev(struct vm_area_struct *vma,
> +	struct vm_area_struct *pvma, struct vm_area_struct *prev,
> +	struct vm_area_struct *pvma_prev)
> +{

[ ... ]

> +	/* 3) Fork a new anon_node. */
> +	return anon_node_add_child(parent, vma, true);
> +}

[Severity: High]
Are child anon_nodes ever unlinked from fractal_list?

In anon_node_add_child(), new child nodes are added to their parent's
fractal_list:

mm/rmap.c:anon_node_add_child() {
    ...
    list_add(&child->fractal_list, &rbc_ch->fractal_list);
    ...
}

However, there doesn't seem to be a corresponding list_del() for when the node
is destroyed. When a node is ultimately freed, won't anon_node_free() trigger
its VM_BUG_ON(!list_empty(&anon_nod->fractal_list)), or leave a dangling pointer
in the parent's list?


[Severity: High]
Is the implementation of unlink_anon_nodes() missing from this patchset?

mm/internal.h declares unlink_anon_nodes() to handle VMA teardown:

mm/internal.h:
    void unlink_anon_nodes(struct vm_area_struct *vma);

But the function doesn't seem to be implemented or invoked. Additionally,
vma_set_anon_node() does not add entries to vma->anon_vma_chain.

Without a teardown hook executing, won't the anon_node refcounts never be
decremented, resulting in a permanent memory leak regression?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=5

[RFC PATCH v2 06/13] mm: implement unlink_anon_nodes for ANON_VMA_FRACTAL

[tao]

Detaching an anon_node may require traversing or removing it from the
fractal list, so the lock must be held.

Releasing an anon_node may propagate through the parent chain, while
the actual release can be performed outside the lock.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/rmap.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)

diff --git a/mm/rmap.c b/mm/rmap.c
index 93fa04c5a007..ea3557d3b56c 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -859,6 +859,94 @@ int anon_node_fork_with_prev(struct vm_area_struct *vma,
 	return anon_node_add_child(parent, vma, true);
 }
 
+/* Returns true if untrack succeeds. */
+static bool anon_node_untrack_rmap_locked(struct anon_node *anon_nod,
+		struct vm_area_struct *vma, struct anon_node **release_node)
+{
+	struct anon_node *root = anon_nod->root;
+	struct anon_node *node = anon_nod;
+	struct anon_node *empty_nod = NULL;
+	bool test_empty = false;
+
+	do {
+		if (try_untrack_rmap(node, vma_rmap_base(vma), &test_empty))
+			break;
+		node = anon_node_next_rbc_child(anon_nod, node);
+	} while (node);
+
+	if (!test_empty) {
+		VM_BUG_ON_VMA(!node, vma);
+		return (bool)node;
+	}
+
+	if (node->nr_children == 0)
+		empty_nod = node;
+	else if (node == anon_nod) {
+		/* Replace an rbc child. */
+		node = anon_node_next_rbc_child(anon_nod, anon_nod);
+		if (node && try_reuse_anon_node(anon_nod,
+						atomic_long_read(&node->rbc))) {
+			atomic_long_set(&node->rbc, 0);
+			empty_nod = node;
+		}
+	}
+	/* may be released up the parent chain */
+	*release_node = empty_nod;
+	while (empty_nod && empty_nod != root) {
+		struct anon_node *parent = empty_nod->parent;
+
+		/* delete the empty_nod first, release later. */
+		list_del_init(&empty_nod->fractal_list);
+		empty_nod->mm = NULL;
+		if (--parent->nr_children == 0 && !anon_node_rmap_count(parent))
+			empty_nod = parent;
+		else {
+			empty_nod->parent = NULL;
+			break;
+		}
+	}
+	return true;
+}
+
+static void release_anon_nodes(struct anon_node *anon_nod)
+{
+	struct anon_node *root = anon_nod->root;
+
+	while (anon_nod) {
+		struct anon_node *parent = anon_nod->parent;
+
+		put_anon_node(anon_nod);
+		anon_nod = (anon_nod != root) ? parent : NULL;
+	}
+}
+
+/* Remove link between this VMA and its anon_node. */
+void unlink_anon_nodes(struct vm_area_struct *vma)
+{
+	struct anon_node *anon_nod = vma_anon_node(vma);
+	struct rw_semaphore *rmap_sem;
+	struct anon_node *release_node = NULL;
+
+	/* Always hold mmap lock, read-lock on unmap possibly. */
+	mmap_assert_locked(vma->vm_mm);
+
+	/* Unfaulted */
+	if (!anon_nod)
+		return;
+
+	rmap_sem = anon_node_rmap_sem(anon_nod->root);
+	down_write(rmap_sem);
+	anon_nod = vma_anon_node(vma); /* Recheck after locking. */
+	if (anon_nod) {
+		anon_node_untrack_rmap_locked(anon_nod, vma, &release_node);
+		vma->anon_vma = NULL;
+	}
+	up_write(rmap_sem);
+
+	if (release_node)
+		release_anon_nodes(release_node);
+}
+
 
 #endif
 
-- 
2.17.1

[RFC PATCH v2 07/13] mm: handle rmap_base changes for ANON_VMA_FRACTAL

[tao]

relocate_vma_down() updates vm_start without changing vm_pgoff, which
causes the VMA rmap_base to change.

track_rmap may need memory allocation and requires both old and new
rmap_base, so this cannot be handled inside
anon_vma_interval_tree_pre_update_vma() and
anon_vma_interval_tree_post_update_vma().

Introduce vma_pre_update_rmap_base() and
vma_post_update_rmap_base(), and call them before vma_prepare()
and after vma_complete() in vma_expand(), commit_merge(), and
vma_shrink(), where rmap_base may change.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 10 ++++++++-
 mm/rmap.c     | 60 ++++++++++++++++++++++++++++++++++++++++-----------
 mm/vma.c      | 20 +++++++++++++++--
 3 files changed, 74 insertions(+), 16 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 710baf02575b..970f11b1af50 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -314,7 +314,12 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma);
 int  __anon_vma_prepare(struct vm_area_struct *vma);
 void unlink_anon_vmas(struct vm_area_struct *vma);
 
-#ifdef CONFIG_ANON_VMA_FRACTAL
+#ifndef CONFIG_ANON_VMA_FRACTAL
+static inline int vma_pre_update_rmap_base(struct vm_area_struct *vma,
+	unsigned long diff) { return 0; }
+static inline void vma_post_update_rmap_base(struct vm_area_struct *vma,
+	unsigned long diff) {}
+#else
 int anon_node_clone(struct vm_area_struct *vma, struct vm_area_struct *src,
 	enum vma_operation operation);
 int anon_node_fork_with_prev(struct vm_area_struct *vma,
@@ -322,6 +327,9 @@ int anon_node_fork_with_prev(struct vm_area_struct *vma,
 	struct vm_area_struct *pvma_prev);
 int __anon_node_prepare(struct vm_area_struct *vma);
 void unlink_anon_nodes(struct vm_area_struct *vma);
+/* relocate_vma_down() changes vma_rmap_base. */
+int vma_pre_update_rmap_base(struct vm_area_struct *vma, unsigned long diff);
+void vma_post_update_rmap_base(struct vm_area_struct *vma, unsigned long diff);
 
 #endif
 
diff --git a/mm/rmap.c b/mm/rmap.c
index ea3557d3b56c..fd22576b0f58 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -622,7 +622,7 @@ void __put_anon_node(struct anon_node *anon_nod)
 }
 
 static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
-		struct anon_node *parent, bool is_fork)
+		struct anon_node *parent, unsigned long rmap_base, bool is_fork)
 {
 	struct anon_node *anon_nod;
 
@@ -641,7 +641,7 @@ static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
 	anon_nod->nr_children = 0;
 	INIT_LIST_HEAD(&anon_nod->fractal_list);
 	anon_nod->mm = vma->vm_mm;
-	atomic_long_set(&anon_nod->rbc, vma_rmap_base(vma) + 1);
+	atomic_long_set(&anon_nod->rbc, rmap_base + 1);
 	return anon_nod;
 }
 
@@ -700,7 +700,7 @@ int __anon_node_prepare(struct vm_area_struct *vma)
 			return 0;
 	}
 
-	anon_nod = anon_node_alloc(vma, NULL, false);
+	anon_nod = anon_node_alloc(vma, NULL, rmap_base, false);
 	if (unlikely(!anon_nod))
 		return -ENOMEM;
 
@@ -734,7 +734,7 @@ static inline void vma_set_anon_node(struct vm_area_struct *vma,
 }
 
 static int anon_node_add_child(struct anon_node *anon_nod,
-	struct vm_area_struct *vma, bool is_fork)
+	struct vm_area_struct *vma, unsigned long rmap_base, bool is_fork)
 {
 	struct anon_node *child;
 	struct anon_node *root = anon_nod->root;
@@ -743,7 +743,7 @@ static int anon_node_add_child(struct anon_node *anon_nod,
 
 	if (root->depth == 0)
 		root->depth = 1;
-	child = anon_node_alloc(vma, anon_nod, is_fork);
+	child = anon_node_alloc(vma, anon_nod, rmap_base, is_fork);
 	if (!child)
 		return -ENOMEM;
 
@@ -764,10 +764,9 @@ static int anon_node_add_child(struct anon_node *anon_nod,
 
 /* Returns 0 on success, non-zero on failure. */
 static int anon_node_track_rmap(struct anon_node *anon_nod,
-		struct vm_area_struct *vma)
+		struct vm_area_struct *vma, unsigned long rmap_base)
 {
 	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
-	unsigned long rmap_base = vma_rmap_base(vma);
 	struct anon_node *rbc_ch = anon_nod;
 
 	if (try_track_rmap(anon_nod, rmap_base))
@@ -782,7 +781,7 @@ static int anon_node_track_rmap(struct anon_node *anon_nod,
 	if (rbc_ch)
 		return 0;
 
-	return anon_node_add_child(anon_nod, vma, false);
+	return anon_node_add_child(anon_nod, vma, rmap_base, false);
 }
 
 /* vma clones src anon_node and increments the count. */
@@ -796,7 +795,7 @@ int anon_node_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
 		return 0;
 
 	VM_BUG_ON_VMA(vma->anon_vma != (void *)anon_nod, vma);
-	return anon_node_track_rmap(anon_nod, vma);
+	return anon_node_track_rmap(anon_nod, vma, vma_rmap_base(vma));
 }
 
 static struct anon_node *fork_reuse_ancestor(struct anon_node *parent,
@@ -856,12 +855,13 @@ int anon_node_fork_with_prev(struct vm_area_struct *vma,
 		return 0;
 
 	/* 3) Fork a new anon_node. */
-	return anon_node_add_child(parent, vma, true);
+	return anon_node_add_child(parent, vma, rmap_base, true);
 }
 
 /* Returns true if untrack succeeds. */
 static bool anon_node_untrack_rmap_locked(struct anon_node *anon_nod,
-		struct vm_area_struct *vma, struct anon_node **release_node)
+		struct vm_area_struct *vma, unsigned long rmap_base,
+		struct anon_node **release_node)
 {
 	struct anon_node *root = anon_nod->root;
 	struct anon_node *node = anon_nod;
@@ -869,7 +869,7 @@ static bool anon_node_untrack_rmap_locked(struct anon_node *anon_nod,
 	bool test_empty = false;
 
 	do {
-		if (try_untrack_rmap(node, vma_rmap_base(vma), &test_empty))
+		if (try_untrack_rmap(node, rmap_base, &test_empty))
 			break;
 		node = anon_node_next_rbc_child(anon_nod, node);
 	} while (node);
@@ -938,7 +938,8 @@ void unlink_anon_nodes(struct vm_area_struct *vma)
 	down_write(rmap_sem);
 	anon_nod = vma_anon_node(vma); /* Recheck after locking. */
 	if (anon_nod) {
-		anon_node_untrack_rmap_locked(anon_nod, vma, &release_node);
+		anon_node_untrack_rmap_locked(anon_nod, vma,
+			vma_rmap_base(vma), &release_node);
 		vma->anon_vma = NULL;
 	}
 	up_write(rmap_sem);
@@ -947,6 +948,39 @@ void unlink_anon_nodes(struct vm_area_struct *vma)
 		release_anon_nodes(release_node);
 }
 
+int vma_pre_update_rmap_base(struct vm_area_struct *vma, unsigned long diff)
+{
+	struct anon_node *anon_nod = vma_anon_node(vma);
+	int err;
+
+	vma_assert_write_locked(vma);
+	if (!anon_nod || !diff)
+		return 0;
+
+	err = anon_node_track_rmap(anon_nod, vma, vma_rmap_base(vma) + diff);
+	if (!err)
+		down_write(anon_node_rmap_sem(anon_nod));
+	return err;
+}
+
+void vma_post_update_rmap_base(struct vm_area_struct *vma, unsigned long diff)
+{
+	struct anon_node *anon_nod = vma_anon_node(vma);
+	struct rw_semaphore *rmap_sem;
+	struct anon_node *release_node = NULL;
+
+	vma_assert_write_locked(vma);
+	if (!anon_nod || !diff)
+		return;
+
+	rmap_sem = anon_node_rmap_sem(anon_nod);
+	rwsem_assert_held_write(rmap_sem);
+	anon_node_untrack_rmap_locked(anon_nod, vma,
+		vma_rmap_base(vma) - diff, &release_node);
+	up_write(rmap_sem);
+	if (release_node)
+		release_anon_nodes(release_node);
+}
 
 #endif
 
diff --git a/mm/vma.c b/mm/vma.c
index 9eea2850818a..d2996369e30d 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -732,6 +732,7 @@ static int commit_merge(struct vma_merge_struct *vmg)
 {
 	struct vm_area_struct *vma;
 	struct vma_prepare vp;
+	unsigned long rbase_diff;
 
 	if (vmg->__adjust_next_start) {
 		/* We manipulate middle and adjust next, which is the target. */
@@ -748,12 +749,16 @@ static int commit_merge(struct vma_merge_struct *vmg)
 	/*
 	 * If vmg->give_up_on_oom is set, we're safe, because we don't actually
 	 * manipulate any VMAs until we succeed at preallocation.
-	 *
-	 * Past this point, we will not return an error.
 	 */
 	if (vma_iter_prealloc(vmg->vmi, vma))
 		return -ENOMEM;
 
+	rbase_diff = rmap_base(vmg->start, vmg->pgoff) - vma_rmap_base(vma);
+	if (rbase_diff && vma_pre_update_rmap_base(vma, rbase_diff)) {
+		vma_iter_free(vmg->vmi);
+		return -ENOMEM;
+	}
+
 	vma_prepare(&vp);
 	/*
 	 * THP pages may need to do additional splits if we increase
@@ -766,6 +771,8 @@ static int commit_merge(struct vma_merge_struct *vmg)
 	vma_iter_store_overwrite(vmg->vmi, vmg->target);
 
 	vma_complete(&vp, vmg->vmi, vma->vm_mm);
+	if (rbase_diff)
+		vma_post_update_rmap_base(vma, rbase_diff);
 
 	return 0;
 }
@@ -1248,6 +1255,7 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	       unsigned long start, unsigned long end, pgoff_t pgoff)
 {
 	struct vma_prepare vp;
+	unsigned long rbase_diff;
 
 	WARN_ON((vma->vm_start != start) && (vma->vm_end != end));
 
@@ -1261,6 +1269,12 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
 
 	vma_start_write(vma);
 
+	rbase_diff = rmap_base(start, pgoff) - vma_rmap_base(vma);
+	if (rbase_diff && vma_pre_update_rmap_base(vma, rbase_diff)) {
+		vma_iter_free(vmi);
+		return -ENOMEM;
+	}
+
 	init_vma_prep(&vp, vma);
 	vma_prepare(&vp);
 	vma_adjust_trans_huge(vma, start, end, NULL);
@@ -1268,6 +1282,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	vma_iter_clear(vmi);
 	vma_set_range(vma, start, end, pgoff);
 	vma_complete(&vp, vmi, vma->vm_mm);
+	if (rbase_diff)
+		vma_post_update_rmap_base(vma, rbase_diff);
 	validate_mm(vma->vm_mm);
 	return 0;
 }
-- 
2.17.1

Re: [RFC PATCH v2 07/13] mm: handle rmap_base changes for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Self-deadlock in VMA modification paths due to recursive acquisition of the anon_vma rw_semaphore, followed by unbalanced unlock leading to semaphore corruption.
--

--- Patch [7]: [RFC PATCH v2 07/13] mm: handle rmap_base changes for ANON_VMA_FRACTAL ---
commit 4ece2a0d48ed77392c6b5765afc8e465e5eeb821
Author: tao <tao.wangtao@honor.com>

mm: handle rmap_base changes for ANON_VMA_FRACTAL

This patch introduces vma_pre_update_rmap_base() and
vma_post_update_rmap_base() to track rmap_base changes during VMA merges
and shrinks. It calls these helpers before vma_prepare() and after
vma_complete() in paths where the rmap_base may change.

> diff --git a/mm/vma.c b/mm/vma.c
> index 9eea2850818a8..d2996369e30d6 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
[ ... ]
> @@ -748,12 +749,16 @@ static int commit_merge(struct vma_merge_struct *vmg)
>  	/*
>  	 * If vmg->give_up_on_oom is set, we're safe, because we don't actually
>  	 * manipulate any VMAs until we succeed at preallocation.
>  	 */
>  	if (vma_iter_prealloc(vmg->vmi, vma))
>  		return -ENOMEM;
>  
> +	rbase_diff = rmap_base(vmg->start, vmg->pgoff) - vma_rmap_base(vma);
> +	if (rbase_diff && vma_pre_update_rmap_base(vma, rbase_diff)) {
> +		vma_iter_free(vmg->vmi);
> +		return -ENOMEM;
> +	}
> +
>  	vma_prepare(&vp);

[Severity: High]
Will this sequence cause a self-deadlock on the anon_vma rw_semaphore?

Looking at vma_pre_update_rmap_base(), it acquires the write lock on the
semaphore via down_write(anon_node_rmap_sem(anon_nod)). 

Immediately after that, vma_prepare() is called here. Since vma_prepare()
unconditionally calls anon_vma_lock_write() which attempts to acquire the
exact same rw_semaphore, won't the second down_write() block indefinitely
because Linux rw_semaphores are not recursive for writers?

The exact same pattern is also present in vma_shrink().

> diff --git a/mm/rmap.c b/mm/rmap.c
> index 900a9d9e5e6a3..2a31f0441c114 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
[ ... ]
> +void vma_post_update_rmap_base(struct vm_area_struct *vma, unsigned long diff)
> +{
> +	struct anon_node *anon_nod = vma_anon_node(vma);
> +	struct rw_semaphore *rmap_sem;
> +	struct anon_node *release_node = NULL;
> +
> +	vma_assert_write_locked(vma);
> +	if (!anon_nod || !diff)
> +		return;
> +
> +	rmap_sem = anon_node_rmap_sem(anon_nod);
> +	rwsem_assert_held_write(rmap_sem);
> +	anon_node_untrack_rmap_locked(anon_nod, vma,
> +		vma_rmap_base(vma) - diff, &release_node);
> +	up_write(rmap_sem);

[Severity: High]
Does this up_write() corrupt the semaphore state?

This function is called in commit_merge() and vma_shrink() right after
vma_complete(). Because vma_complete() releases the locks previously
acquired by vma_prepare(), the rw_semaphore has already been unlocked by the
time we reach this code.

Will calling up_write() on an already-unlocked semaphore cause kernel
instability or further lock corruption?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=7

[RFC PATCH v2 08/13] mm: implement anonymous folio rmap for ANON_VMA_FRACTAL

[tao]

Add the ANON_RMAP_FOREACH_VMA() macro to encapsulate rmap traversal and
hide implementation differences from external users.

During reverse mapping of an anonymous folio, traversal first handles the
folio's own anon_node, then iterates through its descendants. Since large
folios may span multiple VMAs, VMA updates are used to advance pgoff, and
pgoff is used to determine completion of the current anon_node traversal.

For KSM, the VMA is located directly using the supplied address.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/rmap.c     | 41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)

diff --git a/mm/internal.h b/mm/internal.h
index 970f11b1af50..b38c7b4c2a8f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -296,6 +296,55 @@ static inline unsigned long anon_node_rmap_count(struct anon_node *anon_nod)
 	return atomic_long_read(&anon_nod->rbc) & ANON_RMAP_BASE_COUNT_MAX;
 }
 
+static inline unsigned long anon_node_rmap_address(struct anon_node *anon_nod,
+		unsigned long pgoff)
+{
+	if (!anon_node_rmap_count(anon_nod))
+		return 0;
+	return anon_node_rmap_base(anon_nod) + (pgoff << PAGE_SHIFT);
+}
+
+static inline struct vm_area_struct *anon_node_lookup_vma(
+	struct anon_node *anon_nod, unsigned long addr, unsigned long pgoff)
+{
+	struct vm_area_struct *vma;
+
+	if (!addr)
+		addr = anon_node_rmap_address(anon_nod, pgoff);
+	if (!anon_nod->mm || !addr)
+		return NULL;
+
+	vma = vma_lookup(anon_nod->mm, addr);
+	return (vma && vma->anon_vma == (void *)anon_nod) ? vma : NULL;
+}
+
+void vm_lock_anon_node(struct mm_struct *mm, struct anon_node *anon_nod);
+void vm_unlock_anon_node(struct anon_node *anon_nod);
+int anon_node_trylock_rmap(struct anon_node *anon_nod);
+void anon_node_lock_rmap(struct anon_node *anon_nod);
+void anon_node_unlock_rmap(struct anon_node *anon_nod);
+
+#define ANON_RMAP_FOREACH_VMA(anon_vma, addr, start, last, rmap_proc_vma) \
+do { \
+	struct vm_area_struct *vma = NULL; /* must be named vma */ \
+	struct anon_node *_anon_nod = (void *)(anon_vma); \
+	struct anon_node *_nod = _anon_nod; \
+	unsigned long _pgoff = (start); \
+	unsigned long _nr = 0, _total = 1; \
+	\
+	while (_nod) { \
+		vma = anon_node_lookup_vma(_nod, (addr), _pgoff); \
+		if (vma) \
+			(rmap_proc_vma); \
+		_pgoff = vma ? vma->vm_pgoff + vma_pages(vma) : _pgoff + 1; \
+		if (!(addr) && _pgoff <= (last)) \
+			continue; \
+		_pgoff = (start); \
+		BUG_ON(++_nr > (_total += _nod->nr_children)); \
+		_nod = anon_node_next_descendant(_anon_nod, _nod); \
+	} \
+} while (0)
+
 #endif
 
 struct anon_vma *folio_get_anon_vma(const struct folio *folio);
diff --git a/mm/rmap.c b/mm/rmap.c
index fd22576b0f58..1551e9d2121d 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -982,6 +982,47 @@ void vma_post_update_rmap_base(struct vm_area_struct *vma, unsigned long diff)
 		release_anon_nodes(release_node);
 }
 
+#define ANON_VM_LOCK_REF	VM_REFCNT_EXCLUDE_READERS_FLAG
+
+void vm_lock_anon_node(struct mm_struct *mm, struct anon_node *anon_nod)
+{
+	atomic_t *vm_lock_ref = anon_node_vm_lock_ref(anon_nod);
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
+
+	if (!(atomic_read(vm_lock_ref) & ANON_VM_LOCK_REF)) {
+		down_write_nest_lock(rmap_sem, &mm->mmap_lock);
+		BUG_ON(atomic_read(vm_lock_ref));
+		atomic_set(vm_lock_ref, ANON_VM_LOCK_REF);
+	}
+}
+
+void vm_unlock_anon_node(struct anon_node *anon_nod)
+{
+	atomic_t *vm_lock_ref = anon_node_vm_lock_ref(anon_nod);
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
+
+	if (atomic_read(vm_lock_ref) & ANON_VM_LOCK_REF) {
+		BUG_ON(atomic_read(vm_lock_ref) != ANON_VM_LOCK_REF);
+		atomic_set(vm_lock_ref, 0);
+		up_write(rmap_sem);
+	}
+}
+
+int anon_node_trylock_rmap(struct anon_node *anon_nod)
+{
+	return down_read_trylock(anon_node_rmap_sem(anon_nod));
+}
+
+void anon_node_lock_rmap(struct anon_node *anon_nod)
+{
+	down_read(anon_node_rmap_sem(anon_nod));
+}
+
+void anon_node_unlock_rmap(struct anon_node *anon_nod)
+{
+	up_read(anon_node_rmap_sem(anon_nod));
+}
+
 #endif
 
 /*
-- 
2.17.1

Re: [RFC PATCH v2 08/13] mm: implement anonymous folio rmap for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] Accessing a dangling `mm_struct` pointer in `anon_node_lookup_vma` causes Use-After-Free (UAF) on the `mm_struct` or its maple tree.
- [High] Lockless access to a `SLAB_TYPESAFE_BY_RCU` VMA after RCU is dropped in `anon_node_lookup_vma` causes Use-After-Free.
- [Medium] ANON_RMAP_FOREACH_VMA macro causes an infinite loop if the caller uses a `continue` statement.
- [Medium] Using 0 as a sentinel address breaks legitimate VMAs mapped at address 0x0.
--

--- Patch [8]: [RFC PATCH v2 08/13] mm: implement anonymous folio rmap for ANON_VMA_FRACTAL ---
commit 6798bdd36311c3a484df289386e32cf89ecdeba9
Author: tao <tao.wangtao@honor.com>

mm: implement anonymous folio rmap for ANON_VMA_FRACTAL

This commit adds the ANON_RMAP_FOREACH_VMA() macro to encapsulate rmap
traversal. It traverses the folio's own anon_node and its descendants, using
VMA updates to advance the page offset. For KSM, the VMA is located directly
using the supplied address.

> diff --git a/mm/internal.h b/mm/internal.h
> index 12473ac84fa4e..5f96b15246bf9 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h

[ ... ]

> @@ -303,6 +303,55 @@ static inline unsigned long anon_node_rmap_count(struct anon_node *anon_nod)
>  	return atomic_long_read(&anon_nod->rbc) & ANON_RMAP_BASE_COUNT_MAX;
>  }
>  
> +static inline unsigned long anon_node_rmap_address(struct anon_node *anon_nod,
> +		unsigned long pgoff)
> +{
> +	if (!anon_node_rmap_count(anon_nod))
> +		return 0;
> +	return anon_node_rmap_base(anon_nod) + (pgoff << PAGE_SHIFT);
> +}
> +
> +static inline struct vm_area_struct *anon_node_lookup_vma(
> +	struct anon_node *anon_nod, unsigned long addr, unsigned long pgoff)
> +{
> +	struct vm_area_struct *vma;
> +
> +	if (!addr)
> +		addr = anon_node_rmap_address(anon_nod, pgoff);
> +	if (!anon_nod->mm || !addr)

[Severity: Medium]
Will this silently ignore valid mappings at address 0x0?

If mmap_min_addr is configured to 0, it seems a valid anonymous VMA could be
explicitly mapped at virtual address 0x0. If this happens, addr will be 0, and
this check will return NULL, causing the valid VMA to be hidden from rmap walks.

> +		return NULL;
> +
> +	vma = vma_lookup(anon_nod->mm, addr);

[Severity: High]
Could anon_nod->mm be a dangling pointer here?

When a VMA is unmapped, if unlink_anon_nodes() skips clearing the mm pointer
for the root anon_node, and the anon_node remains alive because folios are in
the swap cache or pinned by GUP, its mm pointer might dangle after the process
exits and its mm_struct is freed.

If ANON_RMAP_FOREACH_VMA() is called later during reclaim, will it dereference
the freed mm_struct here when calling vma_lookup()?

> +	return (vma && vma->anon_vma == (void *)anon_nod) ? vma : NULL;

[Severity: High]
Is it safe to access vma->anon_vma without RCU protection?

vma_lookup() delegates to mtree_load(), which acquires and then releases the RCU
read lock internally. Since the caller of anon_node_lookup_vma() doesn't hold
the mmap_lock (only the anon_node lock), and no outer RCU critical section is
held, the returned VMA pointer appears to be unprotected.

Because the VMA cache is SLAB_TYPESAFE_BY_RCU, its backing memory can be
unmapped and freed by a concurrent unmap outside of an RCU critical section.
Will dereferencing vma->anon_vma here result in a use-after-free?

> +}
> +
> +void vm_lock_anon_node(struct mm_struct *mm, struct anon_node *anon_nod);
> +void vm_unlock_anon_node(struct anon_node *anon_nod);
> +int anon_node_trylock_rmap(struct anon_node *anon_nod);
> +void anon_node_lock_rmap(struct anon_node *anon_nod);
> +void anon_node_unlock_rmap(struct anon_node *anon_nod);
> +
> +#define ANON_RMAP_FOREACH_VMA(anon_vma, addr, start, last, rmap_proc_vma) \
> +do { \
> +	struct vm_area_struct *vma = NULL; /* must be named vma */ \
> +	struct anon_node *_anon_nod = (void *)(anon_vma); \
> +	struct anon_node *_nod = _anon_nod; \
> +	unsigned long _pgoff = (start); \
> +	unsigned long _nr = 0, _total = 1; \
> +	\
> +	while (_nod) { \
> +		vma = anon_node_lookup_vma(_nod, (addr), _pgoff); \
> +		if (vma) \
> +			(rmap_proc_vma); \

[Severity: Medium]
What happens if the caller provides a block for rmap_proc_vma that contains a
continue statement?

Because the loop advancement logic:

		_pgoff = vma ? vma->vm_pgoff + vma_pages(vma) : _pgoff + 1;
		...
		_nod = anon_node_next_descendant(_anon_nod, _nod);

is located after the execution of rmap_proc_vma, a continue statement inside the
caller's block would jump straight to the while (_nod) condition.

Will this skip the advancement of _nod and _pgoff, causing the macro to infinite
loop on the same node?

> +		_pgoff = vma ? vma->vm_pgoff + vma_pages(vma) : _pgoff + 1; \
> +		if (!(addr) && _pgoff <= (last)) \
> +			continue; \
> +		_pgoff = (start); \
> +		BUG_ON(++_nr > (_total += _nod->nr_children)); \
> +		_nod = anon_node_next_descendant(_anon_nod, _nod); \
> +	} \
> +} while (0)

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=8

[RFC PATCH v2 09/13] mm: prepare anon_node replacement for ANON_VMA_FRACTAL

[tao]

Since this replacement involves large code changes, introduce macros
in vma.c first to isolate code differences and split the changes into
smaller patches.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 16 ++++++++++++++++
 mm/rmap.c     |  2 +-
 mm/vma.c      | 52 +++++++++++++++++++++++++++++++++++++++++----------
 3 files changed, 59 insertions(+), 11 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index b38c7b4c2a8f..6f0ff492a89c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -296,6 +296,22 @@ static inline unsigned long anon_node_rmap_count(struct anon_node *anon_nod)
 	return atomic_long_read(&anon_nod->rbc) & ANON_RMAP_BASE_COUNT_MAX;
 }
 
+static inline bool anon_node_can_share(struct anon_node *anon_nod)
+{
+	return anon_nod->depth == 0 &&
+		anon_node_rmap_count(anon_nod) <= ANON_NODE_SHARE_LIMIT;
+}
+
+static inline void anon_node_verify(struct anon_node *node)
+{
+	struct anon_node *rbc_ch = node;
+
+	WARN_ON_ONCE(anon_node_rmap_count(node) == 0 && node->nr_children == 0);
+	while ((rbc_ch = anon_node_next_rbc_child(node, rbc_ch)) != NULL) {
+		WARN_ON_ONCE(anon_node_rmap_count(rbc_ch) == 0);
+	}
+}
+
 static inline unsigned long anon_node_rmap_address(struct anon_node *anon_nod,
 		unsigned long pgoff)
 {
diff --git a/mm/rmap.c b/mm/rmap.c
index 1551e9d2121d..12c6a1bdff3f 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -690,7 +690,7 @@ int __anon_node_prepare(struct vm_area_struct *vma)
 	mmap_assert_locked(mm);
 	might_sleep();
 
-	anon_nod = NULL; /* Fix find_mergeable_anon_vma(vma) later. */
+	anon_nod = (struct anon_node *)find_mergeable_anon_vma(vma);
 	if (anon_nod) {
 		anon_new = (struct anon_vma *)anon_nod;
 		if (try_track_rmap(anon_nod, rmap_base) &&
diff --git a/mm/vma.c b/mm/vma.c
index d2996369e30d..02178112daa0 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -269,19 +269,23 @@ static void __remove_shared_vm_struct(struct vm_area_struct *vma,
 static void
 anon_vma_interval_tree_pre_update_vma(struct vm_area_struct *vma)
 {
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	struct anon_vma_chain *avc;
 
 	list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
 		anon_vma_interval_tree_remove(avc, &avc->anon_vma->rb_root);
+#endif
 }
 
 static void
 anon_vma_interval_tree_post_update_vma(struct vm_area_struct *vma)
 {
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	struct anon_vma_chain *avc;
 
 	list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
 		anon_vma_interval_tree_insert(avc, &avc->anon_vma->rb_root);
+#endif
 }
 
 /*
@@ -651,10 +655,6 @@ void validate_mm(struct mm_struct *mm)
 
 	mt_validate(&mm->mm_mt);
 	for_each_vma(vmi, vma) {
-#ifdef CONFIG_DEBUG_VM_RB
-		struct anon_vma *anon_vma = vma->anon_vma;
-		struct anon_vma_chain *avc;
-#endif
 		unsigned long vmi_start, vmi_end;
 		bool warn = 0;
 
@@ -675,12 +675,19 @@ void validate_mm(struct mm_struct *mm)
 			vma_iter_dump_tree(&vmi);
 		}
 
-#ifdef CONFIG_DEBUG_VM_RB
-		if (anon_vma) {
+#if CONFIG_DEBUG_VM_RB
+		if (vma->anon_vma) {
+#ifdef CONFIG_ANON_VMA_FRACTAL
+			anon_node_verify((void *)vma->anon_vma);
+#else
+			struct anon_vma *anon_vma = vma->anon_vma;
+			struct anon_vma_chain *avc;
+
 			anon_vma_lock_read(anon_vma);
 			list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
 				anon_vma_interval_tree_verify(avc);
 			anon_vma_unlock_read(anon_vma);
+#endif
 		}
 #endif
 		/* Check for a infinite loop */
@@ -2027,8 +2034,13 @@ static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old,
 	if (anon_vma_compatible(a, b)) {
 		struct anon_vma *anon_vma = READ_ONCE(old->anon_vma);
 
+#ifdef CONFIG_ANON_VMA_FRACTAL
+		if (anon_vma && anon_node_can_share((void *)anon_vma))
+			return anon_vma;
+#else
 		if (anon_vma && list_is_singular(&old->anon_vma_chain))
 			return anon_vma;
+#endif
 	}
 	return NULL;
 }
@@ -2157,6 +2169,9 @@ static DEFINE_MUTEX(mm_all_locks_mutex);
 
 static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma)
 {
+#ifdef CONFIG_ANON_VMA_FRACTAL
+	vm_lock_anon_node(mm, (void *)anon_vma);
+#else
 	if (!test_bit(0, (unsigned long *) &anon_vma->root->rb_root.rb_root.rb_node)) {
 		/*
 		 * The LSB of head.next can't change from under us
@@ -2176,6 +2191,7 @@ static void vm_lock_anon_vma(struct mm_struct *mm, struct anon_vma *anon_vma)
 				       &anon_vma->root->rb_root.rb_root.rb_node))
 			BUG();
 	}
+#endif
 }
 
 static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping)
@@ -2237,7 +2253,6 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping)
 int mm_take_all_locks(struct mm_struct *mm)
 {
 	struct vm_area_struct *vma;
-	struct anon_vma_chain *avc;
 	VMA_ITERATOR(vmi, mm, 0);
 
 	mmap_assert_write_locked(mm);
@@ -2278,9 +2293,16 @@ int mm_take_all_locks(struct mm_struct *mm)
 	for_each_vma(vmi, vma) {
 		if (signal_pending(current))
 			goto out_unlock;
-		if (vma->anon_vma)
+		if (vma->anon_vma) {
+#ifdef CONFIG_ANON_VMA_FRACTAL
+			vm_lock_anon_vma(mm, vma->anon_vma);
+#else
+			struct anon_vma_chain *avc;
+
 			list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
 				vm_lock_anon_vma(mm, avc->anon_vma);
+#endif
+		}
 	}
 
 	return 0;
@@ -2292,6 +2314,9 @@ int mm_take_all_locks(struct mm_struct *mm)
 
 static void vm_unlock_anon_vma(struct anon_vma *anon_vma)
 {
+#ifdef CONFIG_ANON_VMA_FRACTAL
+	vm_unlock_anon_node((void *)anon_vma);
+#else
 	if (test_bit(0, (unsigned long *) &anon_vma->root->rb_root.rb_root.rb_node)) {
 		/*
 		 * The LSB of head.next can't change to 0 from under
@@ -2310,6 +2335,7 @@ static void vm_unlock_anon_vma(struct anon_vma *anon_vma)
 			BUG();
 		anon_vma_unlock_write(anon_vma);
 	}
+#endif
 }
 
 static void vm_unlock_mapping(struct address_space *mapping)
@@ -2333,16 +2359,22 @@ static void vm_unlock_mapping(struct address_space *mapping)
 void mm_drop_all_locks(struct mm_struct *mm)
 {
 	struct vm_area_struct *vma;
-	struct anon_vma_chain *avc;
 	VMA_ITERATOR(vmi, mm, 0);
 
 	mmap_assert_write_locked(mm);
 	BUG_ON(!mutex_is_locked(&mm_all_locks_mutex));
 
 	for_each_vma(vmi, vma) {
-		if (vma->anon_vma)
+		if (vma->anon_vma) {
+#ifdef CONFIG_ANON_VMA_FRACTAL
+			vm_unlock_anon_vma(vma->anon_vma);
+#else
+			struct anon_vma_chain *avc;
+
 			list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
 				vm_unlock_anon_vma(avc->anon_vma);
+#endif
+		}
 		if (vma->vm_file && vma->vm_file->f_mapping)
 			vm_unlock_mapping(vma->vm_file->f_mapping);
 	}
-- 
2.17.1

Re: [RFC PATCH v2 09/13] mm: prepare anon_node replacement for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Lockless traversal of the shared `fractal_list` in `anon_node_verify()` leads to data races and potential use-after-free.
- [Low] Using `#if CONFIG_DEBUG_VM_RB` instead of `#ifdef` causes `-Wundef` compilation warnings when the config is disabled.
--

--- Patch [9]: [RFC PATCH v2 09/13] mm: prepare anon_node replacement for ANON_VMA_FRACTAL ---
commit 7ac3c97b907c3a50c0e59be4fc5c3a22d3833da9
Author: tao <tao.wangtao@honor.com>
    
mm: prepare anon_node replacement for ANON_VMA_FRACTAL

This commit introduces macros in vma.c to isolate code differences and
split the large code changes into smaller patches for the anon_node
replacement in ANON_VMA_FRACTAL.

> diff --git a/mm/vma.c b/mm/vma.c
> index d2996369e30d6..02178112daa0b 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c

[ ... ]

> @@ -675,12 +675,19 @@ void validate_mm(struct mm_struct *mm)
>  			vma_iter_dump_tree(&vmi);
>  		}
>  
> -#ifdef CONFIG_DEBUG_VM_RB
> -		if (anon_vma) {
> +#if CONFIG_DEBUG_VM_RB

[Severity: Low]
Does this introduce a -Wundef compiler warning when the config is disabled?

Since CONFIG_DEBUG_VM_RB is a boolean option, it is left undefined rather
than set to 0 when disabled. Using #if instead of #ifdef might cause
build failures when compiling with -Werror.

> +		if (vma->anon_vma) {
> +#ifdef CONFIG_ANON_VMA_FRACTAL
> +			anon_node_verify((void *)vma->anon_vma);

[Severity: Medium]
Can this cause a data race or use-after-free by locklessly iterating over
the fractal_list?

In the legacy path below, anon_vma_lock_read() is explicitly acquired
before traversing the list. However, anon_node_verify() iterates over
the shared anon_node tree using anon_node_next_rbc_child() without
taking anon_node_lock_rmap().

Since the anon_node structure is shared across multiple processes after
fork(), could another process concurrently modify the fractal_list while
validate_mm() is traversing it?

> +#else
> +			struct anon_vma *anon_vma = vma->anon_vma;
> +			struct anon_vma_chain *avc;
> +
>  			anon_vma_lock_read(anon_vma);
>  			list_for_each_entry(avc, &vma->anon_vma_chain, same_vma)
>  				anon_vma_interval_tree_verify(avc);
>  			anon_vma_unlock_read(anon_vma);
> +#endif
>  		}
>  #endif

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=9

[RFC PATCH v2 10/13] mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL

[tao]

Replace anon_vma with anon_node while keeping existing APIs to
minimize code changes.

Remove anon_vma_chain from VMA to save 16 bytes on 64-bit.

Use ANON_RMAP_FOREACH_VMA for anonymous reverse mapping walks in
ksm.c, memory-failure.c, and rmap.c.

Keep CONFIG_ANON_VMA_FRACTAL enabling in a separate patch for
easier control.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 include/linux/mm.h       |  2 ++
 include/linux/mm_types.h |  2 ++
 include/linux/rmap.h     |  5 +++-
 mm/internal.h            | 58 +++++++++++++++++++++++++++++++++++-----
 mm/ksm.c                 | 19 ++++---------
 mm/memory-failure.c      |  8 ++----
 mm/mmap.c                |  7 ++---
 mm/rmap.c                | 43 +++++++++++++++--------------
 mm/vma.c                 |  4 +++
 mm/vma_init.c            |  2 ++
 10 files changed, 100 insertions(+), 50 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbdd..3fc2f92040b0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -970,7 +970,9 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	memset(vma, 0, sizeof(*vma));
 	vma->vm_mm = mm;
 	vma->vm_ops = &vma_dummy_vm_ops;
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
+#endif
 	vma_lock_init(vma, false);
 }
 
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index b18c2b2e7d2c..b98090e6cfdd 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -970,8 +970,10 @@ struct vm_area_struct {
 	 * can only be in the i_mmap tree.  An anonymous MAP_PRIVATE, stack
 	 * or brk vma (with NULL file) can only be in an anon_vma list.
 	 */
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	struct list_head anon_vma_chain; /* Serialized by mmap_lock &
 					  * page_table_lock */
+#endif
 	struct anon_vma *anon_vma;	/* Serialized by page_table_lock */
 
 	/* Function pointers to deal with this struct. */
diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 75738f4ede1f..81ad29f3c392 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -15,6 +15,7 @@
 #include <linux/memremap.h>
 #include <linux/bit_spinlock.h>
 
+#ifndef CONFIG_ANON_VMA_FRACTAL
 /*
  * The anon_vma heads a list of private "related" vmas, to scan if
  * an anonymous page pointing to this anon_vma needs to be unmapped:
@@ -67,7 +68,9 @@ struct anon_vma {
 	struct rb_root_cached rb_root;
 };
 
-#ifdef CONFIG_ANON_VMA_FRACTAL
+#else
+
+#define anon_node anon_vma
 
 /* struct anon_node - Replace anon_vma for anonymous page reverse mapping. */
 struct anon_node {
diff --git a/mm/internal.h b/mm/internal.h
index 6f0ff492a89c..205f1b5226aa 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -212,6 +212,7 @@ static inline unsigned long vma_rmap_base(const struct vm_area_struct *vma)
 	return rmap_base(vma->vm_start, vma->vm_pgoff);
 }
 
+#ifndef CONFIG_ANON_VMA_FRACTAL
 static inline void get_anon_vma(struct anon_vma *anon_vma)
 {
 	atomic_inc(&anon_vma->refcount);
@@ -255,7 +256,20 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
 	up_read(&anon_vma->root->rwsem);
 }
 
-#ifdef CONFIG_ANON_VMA_FRACTAL
+/* Provide unified anon_vma VMA iteration interfaces. */
+#define ANON_RMAP_FOREACH_VMA(anon_vma, addr, start, last, rmap_proc_vma) \
+do { \
+	struct vm_area_struct *vma; /* must be named vma */ \
+	struct rb_root_cached *_rb_root = &(anon_vma)->rb_root; \
+	struct anon_vma_chain *_avc; \
+	\
+	anon_vma_interval_tree_foreach(_avc, _rb_root, start, last) { \
+		vma = _avc->vma; \
+		(rmap_proc_vma); \
+	} \
+} while (0)
+
+#else
 
 static inline struct anon_node *anon_node_next_rbc_child(
 		struct anon_node *anon_nod, struct anon_node *node)
@@ -361,6 +375,36 @@ do { \
 	} \
 } while (0)
 
+static inline void get_anon_vma(struct anon_vma *anon_vma)
+{
+	get_anon_node((void *)anon_vma);
+}
+
+static inline void put_anon_vma(struct anon_vma *anon_vma)
+{
+	put_anon_node((void *)anon_vma);
+}
+
+/* Huge pages are protected by folio_lock(), these APIs no longer needed. */
+static inline void anon_vma_lock_write(struct anon_vma *anon_vma) {}
+static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) { return 1; }
+static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) {}
+
+/* These are still used as rmap lock APIs. */
+static inline void anon_vma_lock_read(struct anon_vma *anon_vma)
+{
+	anon_node_lock_rmap((void *)anon_vma);
+}
+
+static inline int anon_vma_trylock_read(struct anon_vma *anon_vma)
+{
+	return anon_node_trylock_rmap((void *)anon_vma);
+}
+
+static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
+{
+	anon_node_unlock_rmap((void *)anon_vma);
+}
 #endif
 
 struct anon_vma *folio_get_anon_vma(const struct folio *folio);
@@ -380,18 +424,20 @@ int  __anon_vma_prepare(struct vm_area_struct *vma);
 void unlink_anon_vmas(struct vm_area_struct *vma);
 
 #ifndef CONFIG_ANON_VMA_FRACTAL
+static inline int anon_vma_fork_with_prev(struct vm_area_struct *vma,
+	struct vm_area_struct *pvma, struct vm_area_struct *vma_prev,
+	struct vm_area_struct *pvma_prev)
+{
+	return anon_vma_fork(vma, pvma);
+}
 static inline int vma_pre_update_rmap_base(struct vm_area_struct *vma,
 	unsigned long diff) { return 0; }
 static inline void vma_post_update_rmap_base(struct vm_area_struct *vma,
 	unsigned long diff) {}
 #else
-int anon_node_clone(struct vm_area_struct *vma, struct vm_area_struct *src,
-	enum vma_operation operation);
-int anon_node_fork_with_prev(struct vm_area_struct *vma,
+int anon_vma_fork_with_prev(struct vm_area_struct *vma,
 	struct vm_area_struct *pvma, struct vm_area_struct *vma_prev,
 	struct vm_area_struct *pvma_prev);
-int __anon_node_prepare(struct vm_area_struct *vma);
-void unlink_anon_nodes(struct vm_area_struct *vma);
 /* relocate_vma_down() changes vma_rmap_base. */
 int vma_pre_update_rmap_base(struct vm_area_struct *vma, unsigned long diff);
 void vma_post_update_rmap_base(struct vm_area_struct *vma, unsigned long diff);
diff --git a/mm/ksm.c b/mm/ksm.c
index 7d5b76478f0b..176b4da494ea 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -3174,8 +3174,6 @@ void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc)
 		/* Ignore the stable/unstable/sqnr flags */
 		const unsigned long addr = rmap_item->address & PAGE_MASK;
 		struct anon_vma *anon_vma = rmap_item->anon_vma;
-		struct anon_vma_chain *vmac;
-		struct vm_area_struct *vma;
 
 		cond_resched();
 		if (!anon_vma_trylock_read(anon_vma)) {
@@ -3186,11 +3184,8 @@ void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc)
 			anon_vma_lock_read(anon_vma);
 		}
 
-		anon_vma_interval_tree_foreach(vmac, &anon_vma->rb_root,
-					       0, ULONG_MAX) {
-
+		ANON_RMAP_FOREACH_VMA(anon_vma, addr, 0, ULONG_MAX, ({
 			cond_resched();
-			vma = vmac->vma;
 
 			if (addr < vma->vm_start || addr >= vma->vm_end)
 				continue;
@@ -3214,7 +3209,7 @@ void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc)
 				anon_vma_unlock_read(anon_vma);
 				return;
 			}
-		}
+		}));
 		anon_vma_unlock_read(anon_vma);
 	}
 	if (!search_new_forks++)
@@ -3230,7 +3225,6 @@ void collect_procs_ksm(const struct folio *folio, const struct page *page,
 {
 	struct ksm_stable_node *stable_node;
 	struct ksm_rmap_item *rmap_item;
-	struct vm_area_struct *vma;
 	struct task_struct *tsk;
 
 	stable_node = folio_stable_node(folio);
@@ -3242,22 +3236,19 @@ void collect_procs_ksm(const struct folio *folio, const struct page *page,
 		anon_vma_lock_read(av);
 		rcu_read_lock();
 		for_each_process(tsk) {
-			struct anon_vma_chain *vmac;
 			unsigned long addr;
 			struct task_struct *t =
 				task_early_kill(tsk, force_early);
 			if (!t)
 				continue;
-			anon_vma_interval_tree_foreach(vmac, &av->rb_root, 0,
-						       ULONG_MAX)
-			{
-				vma = vmac->vma;
+			ANON_RMAP_FOREACH_VMA(av, rmap_item->address,
+					0, ULONG_MAX, ({
 				if (vma->vm_mm == t->mm) {
 					addr = rmap_item->address & PAGE_MASK;
 					add_to_kill_ksm(t, page, vma, to_kill,
 							addr);
 				}
-			}
+			}));
 		}
 		rcu_read_unlock();
 		anon_vma_unlock_read(av);
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 51508a55c405..1f44e5f76a42 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -545,21 +545,17 @@ static void collect_procs_anon(const struct folio *folio,
 	pgoff = page_pgoff(folio, page);
 	rcu_read_lock();
 	for_each_process(tsk) {
-		struct vm_area_struct *vma;
-		struct anon_vma_chain *vmac;
 		struct task_struct *t = task_early_kill(tsk, force_early);
 		unsigned long addr;
 
 		if (!t)
 			continue;
-		anon_vma_interval_tree_foreach(vmac, &av->rb_root,
-					       pgoff, pgoff) {
-			vma = vmac->vma;
+		ANON_RMAP_FOREACH_VMA(av, 0, pgoff, pgoff, ({
 			if (vma->vm_mm != t->mm)
 				continue;
 			addr = page_mapped_in_vma(page, vma);
 			add_to_kill_anon_file(t, page, vma, to_kill, addr);
-		}
+		}));
 	}
 	rcu_read_unlock();
 	anon_vma_unlock_read(av);
diff --git a/mm/mmap.c b/mm/mmap.c
index 2311ae7c2ff4..c283e0de819a 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1730,7 +1730,8 @@ bool mmap_read_lock_maybe_expand(struct mm_struct *mm,
 
 __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 {
-	struct vm_area_struct *mpnt, *tmp;
+	struct vm_area_struct *mpnt = NULL, *tmp = NULL;
+	struct vm_area_struct *prev = NULL, *tmp_prev = NULL;
 	int retval;
 	unsigned long charge = 0;
 	LIST_HEAD(uf);
@@ -1759,7 +1760,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 		goto out;
 
 	mt_clear_in_rcu(vmi.mas.tree);
-	for_each_vma(vmi, mpnt) {
+	for ( ; (mpnt = vma_next(&vmi)) != NULL; prev = mpnt, tmp_prev = tmp) {
 		struct file *file;
 
 		retval = vma_start_write_killable(mpnt);
@@ -1800,7 +1801,7 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 			 * copy page for current vma.
 			 */
 			tmp->anon_vma = NULL;
-		} else if (anon_vma_fork(tmp, mpnt))
+		} else if (anon_vma_fork_with_prev(tmp, mpnt, tmp_prev, prev))
 			goto fail_nomem_anon_vma_fork;
 		vm_flags_clear(tmp, VM_LOCKED_MASK);
 		/*
diff --git a/mm/rmap.c b/mm/rmap.c
index 12c6a1bdff3f..11239838b722 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -84,6 +84,7 @@
 #include "internal.h"
 #include "swap.h"
 
+#ifndef CONFIG_ANON_VMA_FRACTAL
 static struct kmem_cache *anon_vma_cachep;
 static struct kmem_cache *anon_vma_chain_cachep;
 
@@ -558,7 +559,16 @@ void __init anon_vma_init(void)
 			SLAB_PANIC|SLAB_ACCOUNT);
 }
 
-#ifdef CONFIG_ANON_VMA_FRACTAL
+void __put_anon_vma(struct anon_vma *anon_vma)
+{
+	struct anon_vma *root = anon_vma->root;
+
+	anon_vma_free(anon_vma);
+	if (root != anon_vma && atomic_dec_and_test(&root->refcount))
+		anon_vma_free(root);
+}
+
+#else
 
 static struct kmem_cache *anon_node_cachep;
 
@@ -581,7 +591,7 @@ static void anon_node_ctor(void *data)
 	atomic_set(&anon_nod->refcount, 0);
 }
 
-static inline void anon_vma_fractal_init(void)
+void __init anon_vma_init(void)
 {
 	struct kmem_cache_args args = {
 		.use_freeptr_offset = true,
@@ -621,6 +631,11 @@ void __put_anon_node(struct anon_node *anon_nod)
 		anon_node_free(root);
 }
 
+static inline void __put_anon_vma(struct anon_vma *anon_vma)
+{
+	__put_anon_node((struct anon_node *)(anon_vma));
+}
+
 static inline struct anon_node *anon_node_alloc(struct vm_area_struct *vma,
 		struct anon_node *parent, unsigned long rmap_base, bool is_fork)
 {
@@ -679,7 +694,7 @@ static bool try_reuse_anon_node(struct anon_node *anon_nod,
 }
 
 /* attach an anon_node to the VMA. */
-int __anon_node_prepare(struct vm_area_struct *vma)
+int __anon_vma_prepare(struct vm_area_struct *vma)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long rmap_base = vma_rmap_base(vma);
@@ -785,7 +800,7 @@ static int anon_node_track_rmap(struct anon_node *anon_nod,
 }
 
 /* vma clones src anon_node and increments the count. */
-int anon_node_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
+int anon_vma_clone(struct vm_area_struct *vma,  struct vm_area_struct *src,
 	enum vma_operation operation)
 {
 	struct anon_node *anon_nod = vma_anon_node(src);
@@ -827,7 +842,7 @@ static struct anon_node *fork_reuse_ancestor(struct anon_node *parent,
 }
 
 /* Clone an anon_node from prev or fork one from the parent pvma. */
-int anon_node_fork_with_prev(struct vm_area_struct *vma,
+int anon_vma_fork_with_prev(struct vm_area_struct *vma,
 	struct vm_area_struct *pvma, struct vm_area_struct *prev,
 	struct vm_area_struct *pvma_prev)
 {
@@ -921,7 +936,7 @@ static void release_anon_nodes(struct anon_node *anon_nod)
 }
 
 /* Remove link between this VMA and its anon_node. */
-void unlink_anon_nodes(struct vm_area_struct *vma)
+void unlink_anon_vmas(struct vm_area_struct *vma)
 {
 	struct anon_node *anon_nod = vma_anon_node(vma);
 	struct rw_semaphore *rmap_sem;
@@ -3377,15 +3392,6 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 EXPORT_SYMBOL_GPL(make_device_exclusive);
 #endif
 
-void __put_anon_vma(struct anon_vma *anon_vma)
-{
-	struct anon_vma *root = anon_vma->root;
-
-	anon_vma_free(anon_vma);
-	if (root != anon_vma && atomic_dec_and_test(&root->refcount))
-		anon_vma_free(root);
-}
-
 static struct anon_vma *rmap_walk_anon_lock(const struct folio *folio,
 					    struct rmap_walk_control *rwc)
 {
@@ -3433,7 +3439,6 @@ static void rmap_walk_anon(struct folio *folio,
 {
 	struct anon_vma *anon_vma;
 	pgoff_t pgoff_start, pgoff_end;
-	struct anon_vma_chain *avc;
 
 	/*
 	 * The folio lock ensures that folio->mapping can't be changed under us
@@ -3453,9 +3458,7 @@ static void rmap_walk_anon(struct folio *folio,
 
 	pgoff_start = folio_pgoff(folio);
 	pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
-	anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root,
-			pgoff_start, pgoff_end) {
-		struct vm_area_struct *vma = avc->vma;
+	ANON_RMAP_FOREACH_VMA(anon_vma, 0, pgoff_start, pgoff_end, ({
 		unsigned long address = vma_address(vma, pgoff_start,
 				folio_nr_pages(folio));
 
@@ -3469,7 +3472,7 @@ static void rmap_walk_anon(struct folio *folio,
 			break;
 		if (rwc->done && rwc->done(folio))
 			break;
-	}
+	}));
 
 	if (!locked)
 		anon_vma_unlock_read(anon_vma);
diff --git a/mm/vma.c b/mm/vma.c
index 02178112daa0..76294a7201ca 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -78,7 +78,11 @@ static bool vma_is_fork_child(struct vm_area_struct *vma)
 	 * parents. This can improve scalability caused by the anon_vma root
 	 * lock.
 	 */
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	return vma && vma->anon_vma && !list_is_singular(&vma->anon_vma_chain);
+#else
+	return vma && vma->anon_vma && vma->anon_vma->depth > 1;
+#endif
 }
 
 static inline bool is_mergeable_vma(struct vma_merge_struct *vmg, bool merge_next)
diff --git a/mm/vma_init.c b/mm/vma_init.c
index 3c0b65950510..ffb7359ea47f 100644
--- a/mm/vma_init.c
+++ b/mm/vma_init.c
@@ -134,7 +134,9 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 		return NULL;
 	}
 	vma_lock_init(new, true);
+#ifndef CONFIG_ANON_VMA_FRACTAL
 	INIT_LIST_HEAD(&new->anon_vma_chain);
+#endif
 	vma_numab_state_init(new);
 	dup_anon_vma_name(orig, new);
 
-- 
2.17.1

Re: [RFC PATCH v2 10/13] mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Desynchronization of `prev` and `tmp_prev` pointers in `dup_mmap()` leads to `anon_node` tree corruption when VMAs are skipped, causing page use-after-free and structural corruption.
- [Critical] The `ANON_RMAP_FOREACH_VMA` macro enters an infinite loop when callers use a `continue` statement, leading to a hard lockup.
- [High] Stubbing `anon_vma_lock_write` to a no-op breaks locking in `mremap` and `khugepaged`, leading to concurrent page table corruption.
--

--- Patch [10]: [RFC PATCH v2 10/13] mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL ---
commit 1deef9c4b5a4ec2adb2fbed6a3ce6762667d351c
Author: tao <tao.wangtao@honor.com>
Subject: mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL

This commit replaces anon_vma with anon_node for anonymous reverse mapping
walks, aiming to save memory by removing anon_vma_chain from VMA. It
introduces a new ANON_RMAP_FOREACH_VMA macro and stubs out older locking
APIs.

> diff --git a/mm/internal.h b/mm/internal.h
> index 1c29f98b1b252..7c34ff1abf85c 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
[ ... ]
> +/* Huge pages are protected by folio_lock(), these APIs no longer needed. */
> +static inline void anon_vma_lock_write(struct anon_vma *anon_vma) {}
> +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) { return 1; }
> +static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) {}

[Severity: High]
Does stubbing out anon_vma_lock_write() leave page table modifications
unprotected against concurrent reverse mapping accesses?

Operations like take_rmap_locks() in mm/mremap.c and collapse_huge_page()
in mm/khugepaged.c still rely on this lock to serialize page table changes.
Without it, there could be page table corruption when try_to_unmap()
locklessly scans them.

> diff --git a/mm/mmap.c b/mm/mmap.c
> index 2311ae7c2ff45..c283e0de819af 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
[ ... ]
>  	mt_clear_in_rcu(vmi.mas.tree);
> -	for_each_vma(vmi, mpnt) {
> +	for ( ; (mpnt = vma_next(&vmi)) != NULL; prev = mpnt, tmp_prev = tmp) {
>  		struct file *file;

[Severity: Critical]
Does this loop advancement logic cause prev and tmp_prev to become
desynchronized if a VMA is skipped?

If mpnt->vm_flags & VM_DONTCOPY is true, the loop in dup_mmap() executes a
continue statement. The step expression will update prev to mpnt (the skipped
VMA), but tmp_prev will remain pointing to the clone of an unrelated older
VMA.

[ ... ]
>  			tmp->anon_vma = NULL;
> -		} else if (anon_vma_fork(tmp, mpnt))
> +		} else if (anon_vma_fork_with_prev(tmp, mpnt, tmp_prev, prev))
>  			goto fail_nomem_anon_vma_fork;

[Severity: Critical]
If prev and tmp_prev are desynchronized due to a prior skipped VMA, will
this cause anon_vma_fork_with_prev() to incorrectly reuse an anon_node from
an unrelated mapping?

This could place child VMAs in the wrong reverse mapping tree, leading to
use-after-free regressions when the kernel attempts to unmap or swap out
pages.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index aa6632790d9af..d3677515fae92 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
[ ... ]
> @@ -3455,9 +3460,7 @@ static void rmap_walk_anon(struct folio *folio,
>  
>  	pgoff_start = folio_pgoff(folio);
>  	pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
> -	anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root,
> -			pgoff_start, pgoff_end) {
> -		struct vm_area_struct *vma = avc->vma;
> +	ANON_RMAP_FOREACH_VMA(anon_vma, 0, pgoff_start, pgoff_end, ({
>  		unsigned long address = vma_address(vma, pgoff_start,
>  				folio_nr_pages(folio));
>  

[Severity: Critical]
Can the use of continue inside this macro block bypass the iterator
advancement logic of ANON_RMAP_FOREACH_VMA?

When CONFIG_ANON_VMA_FRACTAL is enabled, ANON_RMAP_FOREACH_VMA is defined
as a while loop that advances the iterator at the end of the block. If a
caller like rmap_walk_anon() jumps to the next iteration using continue,
it will skip the advancement and result in an infinite loop and softlockup.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=10

[RFC PATCH v2 11/13] mm: optimize rmap for ANON_VMA_FRACTAL with PVL

[tao]

Leaf anon_nodes have no children, and a given address can belong to at
most one VMA. In this case, PVL can protect the rmap.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/internal.h | 22 ++++++++++++++++++
 mm/rmap.c     | 64 ++++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 83 insertions(+), 3 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 205f1b5226aa..f8dd2120b780 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -354,6 +354,23 @@ int anon_node_trylock_rmap(struct anon_node *anon_nod);
 void anon_node_lock_rmap(struct anon_node *anon_nod);
 void anon_node_unlock_rmap(struct anon_node *anon_nod);
 
+#define ANON_RMAP_LEAF_VMA	1
+#define ANON_RMAP_FLAGS 	(ANON_RMAP_LEAF_VMA)
+
+static inline bool anon_rmap_is_leaf_vma(void *anon_rmap)
+{
+#ifdef CONFIG_PER_VMA_LOCK
+	return ((unsigned long)anon_rmap & ANON_RMAP_FLAGS) == ANON_RMAP_LEAF_VMA;
+#else
+	return false;
+#endif
+}
+
+static inline struct vm_area_struct *anon_rmap_to_leaf_vma(void *anon_rmap)
+{
+	return (void *)((unsigned long)anon_rmap & ~ANON_RMAP_FLAGS);
+}
+
 #define ANON_RMAP_FOREACH_VMA(anon_vma, addr, start, last, rmap_proc_vma) \
 do { \
 	struct vm_area_struct *vma = NULL; /* must be named vma */ \
@@ -362,6 +379,11 @@ do { \
 	unsigned long _pgoff = (start); \
 	unsigned long _nr = 0, _total = 1; \
 	\
+	if (anon_rmap_is_leaf_vma(_anon_nod)) { \
+		vma = anon_rmap_to_leaf_vma(_anon_nod); \
+		(rmap_proc_vma); \
+		break; \
+	} \
 	while (_nod) { \
 		vma = anon_node_lookup_vma(_nod, (addr), _pgoff); \
 		if (vma) \
diff --git a/mm/rmap.c b/mm/rmap.c
index 11239838b722..d1099a2fef0a 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -572,6 +572,8 @@ void __put_anon_vma(struct anon_vma *anon_vma)
 
 static struct kmem_cache *anon_node_cachep;
 
+
+
 static inline struct rw_semaphore *anon_node_rmap_sem(struct anon_node *node)
 {
 	return &node->root->rwsem;
@@ -1035,9 +1037,48 @@ void anon_node_lock_rmap(struct anon_node *anon_nod)
 
 void anon_node_unlock_rmap(struct anon_node *anon_nod)
 {
+#ifdef CONFIG_PER_VMA_LOCK
+	/* Corresponds to folio_lock_anon_vma_read(). */
+	if (anon_rmap_is_leaf_vma(anon_nod)) {
+		vma_refcount_put(anon_rmap_to_leaf_vma(anon_nod));
+		return;
+	}
+#endif
+
 	up_read(anon_node_rmap_sem(anon_nod));
 }
 
+static struct vm_area_struct *trylock_leaf_vma_rmap(
+		struct anon_node *anon_nod, const struct folio *folio,
+		struct mm_struct **other_mm)
+{
+	unsigned long pgoff_start = folio_pgoff(folio);
+	unsigned long pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
+	unsigned long addr = anon_node_rmap_address(anon_nod, pgoff_start);
+	struct mm_struct *mm = anon_nod->mm;
+	struct vm_area_struct *vma;
+	int oldcnt;
+
+	if (anon_nod->nr_children)
+		return NULL;
+
+	if (!addr || !mm || !mmget_not_zero(mm))
+		return NULL;
+
+	vma = vma_lookup(mm, addr);
+	mmput_async(mm);
+	if (!vma || vma_last_pgoff(vma) < pgoff_end)
+		return NULL;
+
+	if (!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
+						     VM_REFCNT_LIMIT)) {
+		return NULL;
+	}
+
+	*other_mm = (vma->vm_mm == mm) ? NULL : vma->vm_mm;
+	return vma;
+}
+
 #endif
 
 /*
@@ -1117,6 +1158,8 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 {
 	struct anon_vma *anon_vma = NULL;
 	struct anon_vma *root_anon_vma;
+	struct vm_area_struct *leaf_vma = NULL;
+	struct mm_struct *other_mm = NULL;
 	unsigned long anon_mapping;
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -1130,6 +1173,15 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 
 	anon_vma = (struct anon_vma *) (anon_mapping - FOLIO_MAPPING_ANON);
 	root_anon_vma = READ_ONCE(anon_vma->root);
+#if defined(CONFIG_PER_VMA_LOCK) && defined(CONFIG_ANON_VMA_FRACTAL)
+	leaf_vma = trylock_leaf_vma_rmap(anon_vma, folio, &other_mm);
+	if (leaf_vma && !other_mm && folio_mapped(folio)) {
+		rcu_read_unlock();
+		return (void *)leaf_vma + ANON_RMAP_LEAF_VMA;
+	}
+	if (unlikely(other_mm))
+		mmgrab(other_mm); /* put vma after unlock. */
+#endif
 	if (down_read_trylock(&root_anon_vma->rwsem)) {
 		/*
 		 * If the folio is still mapped, then this anon_vma is still
@@ -1156,9 +1208,8 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 	}
 
 	if (!folio_mapped(folio)) {
-		rcu_read_unlock();
 		put_anon_vma(anon_vma);
-		return NULL;
+		goto out;
 	}
 
 	/* we pinned the anon_vma, its safe to sleep */
@@ -1176,10 +1227,17 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 		anon_vma = NULL;
 	}
 
-	return anon_vma;
+	goto out_unlocked;
 
 out:
 	rcu_read_unlock();
+out_unlocked:
+#ifdef CONFIG_PER_VMA_LOCK
+	if (leaf_vma)
+		vma_refcount_put(leaf_vma);
+#endif
+	if (other_mm)
+		mmdrop(other_mm);
 	return anon_vma;
 }
 
-- 
2.17.1

Re: [RFC PATCH v2 11/13] mm: optimize rmap for ANON_VMA_FRACTAL with PVL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
- [High] Use-After-Free of `mm_struct` in `vma_refcount_put()` due to dropped `mm` pin.
- [High] Per-VMA lock `vm_lock_seq` exclusion bypass in `trylock_leaf_vma_rmap()`.
- [High] Lockless speculative access of `anon_nod->mm` risks dereferencing a freed pointer.
- [High] Use-After-Free of `anon_vma` on `folio_lock_anon_vma_read()` error path.
- [High] `trylock_leaf_vma_rmap()` locks completely unrelated VMAs due to missing `anon_vma` validation.
- [High] TOCTOU race due to missing VMA bounds re-validation after acquiring the VMA read lock.
- [High] Valid zero-address VMAs silently fail the fast path and are ignored by rmap iteration.
--

--- Patch [11]: [RFC PATCH v2 11/13] mm: optimize rmap for ANON_VMA_FRACTAL with PVL ---
commit e9e971f63165144548b2891ce74a7cd4448a1490
Author: tao <tao.wangtao@honor.com>

mm: optimize rmap for ANON_VMA_FRACTAL with PVL
    
Leaf anon_nodes have no children, and a given address can belong to at
most one VMA. In this case, PVL can protect the rmap.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index d3677515fae92..0967ba2d2a466 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c

[ ... ]

> @@ -1035,9 +1037,48 @@ void anon_node_lock_rmap(struct anon_node *anon_nod)
>  
>  void anon_node_unlock_rmap(struct anon_node *anon_nod)
>  {
> +#ifdef CONFIG_PER_VMA_LOCK
> +	/* Corresponds to folio_lock_anon_vma_read(). */
> +	if (anon_rmap_is_leaf_vma(anon_nod)) {
> +		vma_refcount_put(anon_rmap_to_leaf_vma(anon_nod));

[Severity: High]
Can this cause a use-after-free crash by accessing an unpinned mm_struct?

In trylock_leaf_vma_rmap(), the original mm_struct is temporarily pinned
via mmget_not_zero(mm), but the pin is immediately dropped with
mmput_async(mm). It then checks vma->vm_mm == mm and sets other_mm = NULL.

The caller folio_lock_anon_vma_read() sees other_mm == NULL and returns
the tagged leaf VMA without holding any pin (via mmgrab()) on the
mm_struct. When the rmap operation completes, anon_node_unlock_rmap()
calls vma_refcount_put() here. Since the thread holds no reference to
the mm_struct, it can be freed concurrently. vma_refcount_put() then
accesses the unpinned mm_struct to call
rcuwait_wake_up(&vma->vm_mm->vma_writer_wait), which can crash.

> +		return;
> +	}
> +#endif
> +
>  	up_read(anon_node_rmap_sem(anon_nod));
>  }
>  
> +static struct vm_area_struct *trylock_leaf_vma_rmap(
> +		struct anon_node *anon_nod, const struct folio *folio,
> +		struct mm_struct **other_mm)
> +{
> +	unsigned long pgoff_start = folio_pgoff(folio);
> +	unsigned long pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
> +	unsigned long addr = anon_node_rmap_address(anon_nod, pgoff_start);
> +	struct mm_struct *mm = anon_nod->mm;
> +	struct vm_area_struct *vma;
> +	int oldcnt;
> +
> +	if (anon_nod->nr_children)
> +		return NULL;
> +
> +	if (!addr || !mm || !mmget_not_zero(mm))

[Severity: High]
Is it safe to read anon_nod->mm locklessly under RCU?

The anon_node structure is allocated with SLAB_TYPESAFE_BY_RCU, meaning
the object itself can be freed and reused. However, the mm_struct it
points to is not RCU protected and can be completely freed and returned
to the allocator. Dereferencing the potentially stale mm pointer from
the SLAB-recycled anon_nod and passing it to mmget_not_zero() could
access unmapped memory or corrupt unrelated structures.

[Severity: High]
Does this unintentionally exclude valid VMAs mapped at address zero?

anon_node_rmap_address() can legitimately return 0 for an anonymous VMA
mapped at virtual address 0. Checking if (!addr) treats 0 as an error
rather than a valid address, which silently skips valid zero-address
VMAs for rmap walkers, hwpoison, and KSM.

> +		return NULL;
> +
> +	vma = vma_lookup(mm, addr);
> +	mmput_async(mm);
> +	if (!vma || vma_last_pgoff(vma) < pgoff_end)

[Severity: High]
Could this inadvertently lock a completely unrelated VMA?

This uses vma_lookup() to locate a VMA and returns it if it covers the
target offset, but fails to verify that the returned VMA's anon_vma
pointer matches the anon_nod, or even if the VMA is anonymous. If the
original VMA was unmapped and a completely unrelated VMA was mapped at
the exact same address, vma_lookup() will find it. This unrelated VMA
will be incorrectly locked and returned to the rmap walker, skipping
the true anon_vma tree and executing callbacks on the wrong VMA.

> +		return NULL;
> +
> +	if (!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
> +						     VM_REFCNT_LIMIT)) {

[Severity: High]
Does this bypass the required per-VMA lock vm_lock_seq exclusion?

The per-VMA lock algorithm strictly requires a reader to increment
vm_refcnt and then verify that vma->vm_lock_seq != mm->mm_lock_seq to
ensure no writer is currently locking the VMA. This successfully grants
the VMA read lock purely by incrementing vm_refcnt, failing to perform
any vm_lock_seq exclusion checks. A reader will incorrectly proceed even
if a writer holds the VMA lock, leading to potential data corruption as
the rmap walker accesses page tables concurrently with structural VMA
modifications.

[Severity: High]
Is there a time-of-check to time-of-use race here?

vma_last_pgoff(vma) < pgoff_end is checked locklessly under RCU just
before this. Between that check and the increment of vm_refcnt to
acquire the read lock, a concurrent writer could modify the VMA. Because
this function fails to re-check the VMA bounds after locking, the rmap
walker might receive a locked VMA that is too small and no longer covers
the required address range, resulting in out-of-bounds page table
accesses.

> +		return NULL;
> +	}
> +
> +	*other_mm = (vma->vm_mm == mm) ? NULL : vma->vm_mm;
> +	return vma;
> +}

[ ... ]

> @@ -1156,9 +1208,8 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
>  	}
>  
>  	if (!folio_mapped(folio)) {
> -		rcu_read_unlock();
>  		put_anon_vma(anon_vma);
> -		return NULL;
> +		goto out;

[Severity: High]
Will this leave the caller with a freed anon_vma pointer?

In this fallback path, if down_read_trylock() fails and
!folio_mapped(folio) is true, the code calls put_anon_vma(anon_vma).
The original code correctly returned NULL at this point. By replacing
this with goto out, it falls through to return anon_vma without setting
anon_vma to NULL. The caller receives a non-NULL pointer, falsely
assuming the read lock was acquired, and will operate on the unpinned
and potentially freed anon_vma object.

>  	}
>  
>  	/* we pinned the anon_vma, its safe to sleep */

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=11

[RFC PATCH v2 12/13] mm: shared semaphores for ANON_VMA_FRACTAL

[tao]

Use shared semaphores for anon_node to reduce memory overhead, as rmap
operations rarely run concurrently and fork/delete updates are
short-lived.

To reduce the fork impact of shared semaphores, reuse the root anon_node's
unused parent pointer to store a dedicated semaphore. Revalidate the lock
when acquiring it in rmap paths, since it may change.

Export rwsem_is_write_locked() in rwsem.h, and use it together with
rwsem_is_locked() to detect whether anon_node is in rmap, which can
reduce lock contention.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 include/linux/rmap.h  |   7 ++-
 include/linux/rwsem.h |  10 ++++
 mm/Kconfig            |   9 +++
 mm/rmap.c             | 134 +++++++++++++++++++++++++++++++++++++++---
 4 files changed, 152 insertions(+), 8 deletions(-)

diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 81ad29f3c392..50919c66c368 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -75,11 +75,16 @@ struct anon_vma {
 /* struct anon_node - Replace anon_vma for anonymous page reverse mapping. */
 struct anon_node {
 	struct anon_node *root;		/* Root of this anon_node fractal tree */
+#ifndef CONFIG_ANON_VMA_SHARED_SEMS
 	struct rw_semaphore rwsem;
 	atomic_t vm_lock_ref;		/* for vm_lock_anon_vma */
+#endif
 	atomic_t refcount;
 
-	struct anon_node *parent;	/* NULL if created by a page fault */
+	union {
+		struct anon_node *parent;		/* depth > 1 */
+		struct anon_semaphore *root_anon_sema;	/* depth == 1 */
+	};
 
 	/*
 	 * depth is initialized to 0. When a child is added, depth is set to 1.
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 6a1a7bae5f81..bc50a00a2c00 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -75,6 +75,11 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)
 	return atomic_long_read(&sem->count) != RWSEM_UNLOCKED_VALUE;
 }
 
+static inline int rwsem_is_write_locked(struct rw_semaphore *sem)
+{
+	return atomic_long_read(&sem->count) & RWSEM_WRITER_LOCKED;
+}
+
 static inline void rwsem_assert_held_nolockdep(const struct rw_semaphore *sem)
 	__assumes_ctx_lock(sem)
 {
@@ -181,6 +186,11 @@ static __always_inline int rwsem_is_locked(const struct rw_semaphore *sem)
 	return rw_base_is_locked(&sem->rwbase);
 }
 
+static inline int rwsem_is_write_locked(struct rw_semaphore *sem)
+{
+	return rw_base_is_write_locked(&sem->rwbase);
+}
+
 static __always_inline void rwsem_assert_held_nolockdep(const struct rw_semaphore *sem)
 	__assumes_ctx_lock(sem)
 {
diff --git a/mm/Kconfig b/mm/Kconfig
index c4c6d3e52d72..1b909819035d 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1479,6 +1479,15 @@ config ANON_VMA_FRACTAL
 	  during rmap walks. Since rmap_one operations may switch tasks,
 	  semaphores are still required instead of spinlock_t protection.
 
+config ANON_VMA_SHARED_SEMS
+	bool "anon_vma shared semaphores"
+	def_bool n
+	depends on ANON_VMA_FRACTAL
+	help
+	  As rmap operations rarely run concurrently, and fork/delete updates
+	  are very short, a group of shared semaphores is used to reduce
+	  memory overhead.
+
 config IOMMU_MM_DATA
 	bool
 
diff --git a/mm/rmap.c b/mm/rmap.c
index d1099a2fef0a..3cb516b80eb2 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -559,6 +559,11 @@ void __init anon_vma_init(void)
 			SLAB_PANIC|SLAB_ACCOUNT);
 }
 
+static inline struct rw_semaphore *anon_root_rmap_sem(struct anon_vma *root)
+{
+	return &root->rwsem;
+}
+
 void __put_anon_vma(struct anon_vma *anon_vma)
 {
 	struct anon_vma *root = anon_vma->root;
@@ -571,25 +576,82 @@ void __put_anon_vma(struct anon_vma *anon_vma)
 #else
 
 static struct kmem_cache *anon_node_cachep;
+static struct kmem_cache *anon_sema_cachep;
+
+/* Since rmap operations rarely run concurrently, use shared anon_root_sems. */
+#define ANON_NODE_RMAP_SEMS_BITS	12
+#define ANON_NODE_RMAP_SEMS_COUNT	(1 << ANON_NODE_RMAP_SEMS_BITS)
+
+struct anon_semaphore {
+	struct rw_semaphore sem;
+	atomic_t vm_lock_ref;	/* for vm_lock_anon_vma */
+};
+
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+static struct anon_semaphore anon_node_root_semas[ANON_NODE_RMAP_SEMS_COUNT];
 
+static inline struct anon_semaphore *anon_root_anon_sema(struct anon_node *root)
+{
+	return (root->depth == 1 && root->root_anon_sema) ?
+		root->root_anon_sema :
+		&anon_node_root_semas[hash_ptr(root, ANON_NODE_RMAP_SEMS_BITS)];
+}
+#endif
 
+static inline void anon_sema_init(struct anon_semaphore *anon_sema)
+{
+	init_rwsem(&anon_sema->sem);
+	atomic_set(&anon_sema->vm_lock_ref, 0);
+}
+
+static inline void anon_root_sema_free(struct anon_semaphore *root_sema)
+{
+	VM_BUG_ON(rwsem_is_locked(&root_sema->sem));
+	kmem_cache_free(anon_sema_cachep, root_sema);
+}
+
+static inline struct anon_semaphore *anon_root_sema_alloc(void)
+{
+	struct anon_semaphore *root_sema;
+
+	root_sema = kmem_cache_alloc(anon_sema_cachep, GFP_KERNEL);
+	if (!root_sema)
+		return NULL;
+	anon_sema_init(root_sema);
+	return root_sema;
+}
+
+static inline struct rw_semaphore *anon_root_rmap_sem(struct anon_node *root)
+{
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+	return &anon_root_anon_sema(root)->sem;
+#else
+	return &root->rwsem;
+#endif
+}
 
 static inline struct rw_semaphore *anon_node_rmap_sem(struct anon_node *node)
 {
-	return &node->root->rwsem;
+	return anon_root_rmap_sem(node->root);
 }
 
 static inline atomic_t *anon_node_vm_lock_ref(struct anon_node *node)
 {
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+	return &anon_root_anon_sema(node->root)->vm_lock_ref;
+#else
 	return &node->root->vm_lock_ref;
+#endif
 }
 
 static void anon_node_ctor(void *data)
 {
 	struct anon_node *anon_nod = data;
 
+#ifndef CONFIG_ANON_VMA_SHARED_SEMS
 	init_rwsem(&anon_nod->rwsem);
 	atomic_set(&anon_nod->vm_lock_ref, 0);
+#endif
 	atomic_set(&anon_nod->refcount, 0);
 }
 
@@ -600,10 +662,16 @@ void __init anon_vma_init(void)
 		.freeptr_offset = offsetof(struct anon_node, fractal_list),
 		.ctor = anon_node_ctor,
 	};
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+	int i;
 
+	for (i = 0; i < ANON_NODE_RMAP_SEMS_COUNT; i++)
+		anon_sema_init(&anon_node_root_semas[i]);
+#endif
 	anon_node_cachep = kmem_cache_create("anon_node",
 			sizeof(struct anon_node), &args,
 			SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT);
+	anon_sema_cachep = KMEM_CACHE(anon_semaphore, SLAB_PANIC|SLAB_ACCOUNT);
 }
 
 static inline void anon_node_free(struct anon_node *anon_nod)
@@ -621,6 +689,12 @@ static inline void anon_node_free(struct anon_node *anon_nod)
 		up_write(rmap_sem);
 	}
 
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+	if (anon_nod->depth == 1 && anon_nod->root_anon_sema) {
+		anon_root_sema_free(anon_nod->root_anon_sema);
+		anon_nod->root_anon_sema = NULL;
+	}
+#endif
 	kmem_cache_free(anon_node_cachep, anon_nod);
 }
 
@@ -736,12 +810,17 @@ static inline struct anon_node *vma_anon_node(struct vm_area_struct *vma)
 	return vma ? (struct anon_node *)vma->anon_vma : NULL;
 }
 
+static inline bool no_rmap_reader(struct rw_semaphore *rmap_sem)
+{
+	return !rwsem_is_locked(rmap_sem) || rwsem_is_write_locked(rmap_sem);
+}
+
 static inline void vma_set_anon_node(struct vm_area_struct *vma,
 		struct anon_node *anon_nod, bool is_locked)
 {
 	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(anon_nod);
 
-	if (is_locked) {
+	if (is_locked || no_rmap_reader(rmap_sem)) {
 		vma->anon_vma = (struct anon_vma *)anon_nod;
 		return;
 	}
@@ -758,8 +837,22 @@ static int anon_node_add_child(struct anon_node *anon_nod,
 	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(root);
 	struct anon_node *rbc_ch = anon_nod, *next;
 
-	if (root->depth == 0)
+	if (root->depth == 0) {
+#ifdef CONFIG_ANON_VMA_SHARED_SEMS
+		struct anon_semaphore *root_sema = anon_root_sema_alloc();
+
+		if (!root_sema)
+			return -ENOMEM;
+		down_write(rmap_sem);
+		if (cmpxchg(&root->root_anon_sema, NULL, root_sema) != NULL)
+			anon_root_sema_free(root_sema);
+		root->depth = 1;
+		up_write(rmap_sem);
+#else
 		root->depth = 1;
+#endif
+	}
+
 	child = anon_node_alloc(vma, anon_nod, rmap_base, is_fork);
 	if (!child)
 		return -ENOMEM;
@@ -1025,14 +1118,34 @@ void vm_unlock_anon_node(struct anon_node *anon_nod)
 	}
 }
 
+static struct rw_semaphore *anon_node_lock_rmap_and_verify(
+		struct anon_node *anon_nod, bool trylock)
+{
+	struct anon_node *root = anon_nod->root;
+	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(root);
+
+	while (true) {
+		if (trylock) {
+			if (!down_read_trylock(rmap_sem))
+				return NULL;
+		} else {
+			down_read(rmap_sem);
+		}
+		if (rmap_sem == anon_node_rmap_sem(root))
+			return rmap_sem;
+		up_read(rmap_sem);
+		rmap_sem = anon_node_rmap_sem(root);
+	}
+}
+
 int anon_node_trylock_rmap(struct anon_node *anon_nod)
 {
-	return down_read_trylock(anon_node_rmap_sem(anon_nod));
+	return (bool)anon_node_lock_rmap_and_verify(anon_nod, true);
 }
 
 void anon_node_lock_rmap(struct anon_node *anon_nod)
 {
-	down_read(anon_node_rmap_sem(anon_nod));
+	anon_node_lock_rmap_and_verify(anon_nod, false);
 }
 
 void anon_node_unlock_rmap(struct anon_node *anon_nod)
@@ -1158,6 +1271,7 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 {
 	struct anon_vma *anon_vma = NULL;
 	struct anon_vma *root_anon_vma;
+	struct rw_semaphore *root_rwsem;
 	struct vm_area_struct *leaf_vma = NULL;
 	struct mm_struct *other_mm = NULL;
 	unsigned long anon_mapping;
@@ -1182,14 +1296,20 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 	if (unlikely(other_mm))
 		mmgrab(other_mm); /* put vma after unlock. */
 #endif
-	if (down_read_trylock(&root_anon_vma->rwsem)) {
+trylock_agian:
+	root_rwsem = anon_root_rmap_sem(root_anon_vma);
+	if (down_read_trylock(root_rwsem)) {
+		if (root_rwsem != anon_root_rmap_sem(root_anon_vma)) {
+			up_read(root_rwsem);
+			goto trylock_agian;
+		}
 		/*
 		 * If the folio is still mapped, then this anon_vma is still
 		 * its anon_vma, and holding the mutex ensures that it will
 		 * not go away, see anon_vma_free().
 		 */
 		if (!folio_mapped(folio)) {
-			up_read(&root_anon_vma->rwsem);
+			up_read(root_rwsem);
 			anon_vma = NULL;
 		}
 		goto out;
-- 
2.17.1

Re: [RFC PATCH v2 12/13] mm: shared semaphores for ANON_VMA_FRACTAL

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Lockless RCU readers access freed `anon_semaphore` memory because `anon_sema_cachep` lacks `SLAB_TYPESAFE_BY_RCU`.
- [High] Direct writers and `vm_lock` callers experience TOCTOU races and state loss due to dynamic semaphore upgrades without pointer re-verification.
- [High] An untracked `anon_node` with children retains a dangling `node->mm` pointer to an unmapped VMA's freed `mm_struct`, which KSM rmap walks dereference.
--

--- Patch [12]: [RFC PATCH v2 12/13] mm: shared semaphores for ANON_VMA_FRACTAL ---
commit 415add0f8800d6a7589361c22fc91f31e4dab44c
Author: tao <tao.wangtao@honor.com>

mm: shared semaphores for ANON_VMA_FRACTAL

This commit introduces shared semaphores for anon_node to reduce memory
overhead, reusing the root anon_node's unused parent pointer to store a
dedicated semaphore. It also adds logic to revalidate locks in rmap paths
and uses rwsem_is_write_locked to reduce contention.

> diff --git a/mm/rmap.c b/mm/rmap.c
> index 0967ba2d2a466..196af0ef07ef5 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
[ ... ]
> @@ -600,10 +662,16 @@ void __init anon_vma_init(void)
>  		.freeptr_offset = offsetof(struct anon_node, fractal_list),
>  		.ctor = anon_node_ctor,
>  	};
> +#ifdef CONFIG_ANON_VMA_SHARED_SEMS
> +	int i;
>  
> +	for (i = 0; i < ANON_NODE_RMAP_SEMS_COUNT; i++)
> +		anon_sema_init(&anon_node_root_semas[i]);
> +#endif
>  	anon_node_cachep = kmem_cache_create("anon_node",
>  			sizeof(struct anon_node), &args,
>  			SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT);
> +	anon_sema_cachep = KMEM_CACHE(anon_semaphore, SLAB_PANIC|SLAB_ACCOUNT);
>  }

[Severity: High]
Can lockless RCU readers access freed memory here?

When an anon_node is destroyed, anon_root_sema_free() immediately returns
the semaphore to anon_sema_cachep. Because anon_sema_cachep is created
without SLAB_TYPESAFE_BY_RCU (unlike anon_node_cachep), the memory is
instantly available for reallocation.

If a concurrent lockless RCU reader in folio_lock_anon_vma_read() obtains a
pointer to root_anon_sema just before it is freed:

CPU1
anon_node_free()
  anon_root_sema_free()

CPU2
folio_lock_anon_vma_read()
  root_rwsem = anon_root_rmap_sem(root_anon_vma);
  if (down_read_trylock(root_rwsem)) { ... }

Could CPU2 execute down_read_trylock() on reallocated memory and cause
corruption?

[ ... ]
> @@ -758,8 +837,22 @@ static int anon_node_add_child(struct anon_node *anon_nod,
>  	struct rw_semaphore *rmap_sem = anon_node_rmap_sem(root);
>  	struct anon_node *rbc_ch = anon_nod, *next;
>  
> -	if (root->depth == 0)
> +	if (root->depth == 0) {
> +#ifdef CONFIG_ANON_VMA_SHARED_SEMS
> +		struct anon_semaphore *root_sema = anon_root_sema_alloc();
> +
> +		if (!root_sema)
> +			return -ENOMEM;
> +		down_write(rmap_sem);
> +		if (cmpxchg(&root->root_anon_sema, NULL, root_sema) != NULL)
> +			anon_root_sema_free(root_sema);
> +		root->depth = 1;
> +		up_write(rmap_sem);

[Severity: High]
Are direct writers vulnerable to a TOCTOU race when the semaphore is
upgraded?

This block upgrades the node to use a dedicated semaphore under the shared
write lock. 

If a tree modifier like unlink_anon_vmas() or vma_pre_update_rmap_base()
evaluates anon_node_rmap_sem() and gets the shared semaphore pointer, but
blocks before acquiring it, it might wake up after this upgrade completes. 

Since those paths do not appear to re-verify the pointer like
anon_node_lock_rmap_and_verify() does, will they mistakenly modify the tree
while holding the old shared lock, allowing concurrent modifications from
other threads using the new dedicated lock?

[ ... ]

[Severity: High]
Does this leave a dangling mm pointer on untracked nodes?

When a VMA is unmapped and unlink_anon_vmas() untracks the node,
anon_node_untrack_rmap_locked() skips clearing the mm pointer if the node
still has children.

If the process then exits and frees the mm_struct, could a later KSM rmap
walk pass a non-zero addr to anon_node_lookup_vma(), bypass the untracked
safeguard, and dereference the freed mm_struct in vma_lookup()?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=12

[RFC PATCH v2 13/13] mm: Enable CONFIG_ANON_VMA_FRACTAL by default

[tao]

Everything is now in place to enable CONFIG_ANON_VMA_FRACTAL and
CONFIG_ANON_VMA_SHARED_SEMS for further testing.

Signed-off-by: tao <tao.wangtao@honor.com>
---
 mm/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/Kconfig b/mm/Kconfig
index 1b909819035d..9d4b81bbd4e1 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1453,7 +1453,7 @@ config LOCK_MM_AND_FIND_VMA
 
 config ANON_VMA_FRACTAL
 	bool "anon_vma fractal list"
-	def_bool n
+	def_bool y
 	depends on MMU
 	help
 	  This option uses vma_rmap_base(vma) during rmap to compute the page
@@ -1481,7 +1481,7 @@ config ANON_VMA_FRACTAL
 
 config ANON_VMA_SHARED_SEMS
 	bool "anon_vma shared semaphores"
-	def_bool n
+	def_bool y
 	depends on ANON_VMA_FRACTAL
 	help
 	  As rmap operations rarely run concurrently, and fork/delete updates
-- 
2.17.1

Re: [RFC PATCH v2 13/13] mm: Enable CONFIG_ANON_VMA_FRACTAL by default

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] RMAP walker Use-After-Free of `mm_struct` and VMA during VMA lock release due to skipped `mmgrab()` stabilization and flawed synchronization with `exit_mmap()`.
- [High] Enabling `CONFIG_ANON_VMA_SHARED_SEMS` exposes a Use-After-Free vulnerability because the dynamically allocated semaphores are freed without RCU protection but accessed under RCU.
--

--- Patch [13]: [RFC PATCH v2 13/13] mm: Enable CONFIG_ANON_VMA_FRACTAL by default ---
commit e2497626d5f9a80886b8bd9853568725462fff8d
Author: tao <tao.wangtao@honor.com>

mm: Enable CONFIG_ANON_VMA_FRACTAL by default

Everything is now in place to enable CONFIG_ANON_VMA_FRACTAL and
CONFIG_ANON_VMA_SHARED_SEMS for further testing.

> diff --git a/mm/Kconfig b/mm/Kconfig
> index 1a2c8085b12d5..7916c0127de0a 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -1453,7 +1453,7 @@ config LOCK_MM_AND_FIND_VMA
>  
>  config ANON_VMA_FRACTAL
>  	bool "anon_vma fractal list"
> -	def_bool n
> +	def_bool y

[Severity: High]
This isn't a bug introduced by this patch, but enabling this feature by
default exposes a use-after-free race between the RMAP walker and
exit_mmap().

Looking at trylock_leaf_vma_rmap() in mm/rmap.c, it sets other_mm to NULL if
vma->vm_mm == mm, which skips the mmgrab() stabilization:

mm/rmap.c:trylock_leaf_vma_rmap() {
    ...
    *other_mm = (vma->vm_mm == mm) ? NULL : vma->vm_mm;
    ...
}

Then in folio_lock_anon_vma_read(), RCU protection is dropped while holding
an unpinned leaf_vma:

mm/rmap.c:folio_lock_anon_vma_read() {
    ...
    if (leaf_vma && !other_mm && folio_mapped(folio)) {
        rcu_read_unlock();
        return (void *)leaf_vma + ANON_RMAP_LEAF_VMA;
    }
    ...
}

Concurrently, if exit_mmap() acquires mmap_write_lock() but doesn't drain
per-VMA lock readers via vma_start_write(), it can free the VMA and the
unpinned mm_struct.

When the RMAP walker later calls anon_node_unlock_rmap(), it dereferences
the freed VMA and vm_mm to wake up writers:

mm/rmap.c:anon_node_unlock_rmap() {
    ...
    if (anon_rmap_is_leaf_vma(anon_nod)) {
        vma_refcount_put(anon_rmap_to_leaf_vma(anon_nod));
        return;
    }
    ...
}

Should the RMAP walker ensure the mm_struct is stabilized before dropping
RCU protection to prevent this?

>  	depends on MMU
>  	help
>  	  This option uses vma_rmap_base(vma) during rmap to compute the page
> @@ -1481,7 +1481,7 @@ config ANON_VMA_FRACTAL
>  
>  config ANON_VMA_SHARED_SEMS
>  	bool "anon_vma shared semaphores"
> -	def_bool n
> +	def_bool y

[Severity: High]
This isn't a bug introduced by this patch, but enabling ANON_VMA_SHARED_SEMS
by default exposes a use-after-free vulnerability with the dynamically
allocated semaphores.

In anon_vma_init() in mm/rmap.c, the slab cache for the shared semaphores is
created without SLAB_TYPESAFE_BY_RCU:

mm/rmap.c:anon_vma_init() {
    ...
    anon_sema_cachep = KMEM_CACHE(anon_semaphore, SLAB_PANIC|SLAB_ACCOUNT);
    ...
}

When the semaphore is freed in anon_root_sema_free(), it happens immediately
without an RCU grace period:

mm/rmap.c:anon_root_sema_free() {
    ...
    kmem_cache_free(anon_sema_cachep, root_sema);
    ...
}

However, folio_lock_anon_vma_read() locklessly accesses root->root_anon_sema
under rcu_read_lock() and attempts to modify it:

mm/rmap.c:folio_lock_anon_vma_read() {
    ...
    root_rwsem = anon_root_rmap_sem(root_anon_vma);
    if (down_read_trylock(root_rwsem)) {
    ...
}

Since the semaphore slab is not RCU-safe, the memory can be immediately
reclaimed by the buddy allocator or reallocated. 

Will down_read_trylock() end up modifying freed or completely unrelated
memory if a reclaim runs concurrently with an unmap?

>  	depends on ANON_VMA_FRACTAL
>  	help
>  	  As rmap operations rarely run concurrently, and fork/delete updates

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260707063308.29176-1-tao.wangtao@honor.com?part=13

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[David Hildenbrand (Arm)]

On 7/7/26 08:32, tao wrote:
> Hi all,
> 
> Since v1 was relatively complex and difficult to extend, this version
> adopts a simpler anon_vma implementation approach.

I really hate to repeat myself, but you seemed to have missed my main point [1]

"Lorenzo has been hard at work exploring various design options (and I'm afraid
he might be one of the 3 people on this planet that understand anon_vma in full
detail), so I suggest we wait for a redesign proposal from him and see if that
is doable?"

[1] https://lore.kernel.org/all/70377626-500d-46f2-a3f5-a0d2914ce365@kernel.org/

-- 
Cheers,

David

RE: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[wangtao]

> 
> On 7/7/26 08:32, tao wrote:
> > Hi all,
> >
> > Since v1 was relatively complex and difficult to extend, this version
> > adopts a simpler anon_vma implementation approach.
> 
> I really hate to repeat myself, but you seemed to have missed my main point
> [1]
> 
I saw it, and I’ve waited for a while as well.

If there are technical issues, please point them out directly.
Or is this simply how technical discussions are usually conducted in the Linux community or the mm subsystem?

Thanks,
Tao

> "Lorenzo has been hard at work exploring various design options (and I'm
> afraid he might be one of the 3 people on this planet that understand
> anon_vma in full detail), so I suggest we wait for a redesign proposal from
> him and see if that is doable?"
> 
> [1] https://lore.kernel.org/all/70377626-500d-46f2-a3f5-
> a0d2914ce365@kernel.org/
> 
> --
> Cheers,
> 
> David

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[David Hildenbrand (Arm)]

On 7/7/26 10:21, wangtao wrote:
>>
>> On 7/7/26 08:32, tao wrote:
>>> Hi all,
>>>
>>> Since v1 was relatively complex and difficult to extend, this version
>>> adopts a simpler anon_vma implementation approach.
>>
>> I really hate to repeat myself, but you seemed to have missed my main point
>> [1]
>>
> I saw it, and I’ve waited for a while as well.

Apparently you saw it but you didn't read my reply about slapping more
complexity on something complicated.

 12 files changed, 1056 insertions(+), 54 deletions(-)

> 
> If there are technical issues, please point them out directly.
> Or is this simply how technical discussions are usually conducted in the Linux community or the mm subsystem?

I suggest you go back and read my email.
-- 
Cheers,

David

RE: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[wangtao]

> >> On 7/7/26 08:32, tao wrote:
> >>> Hi all,
> >>>
> >>> Since v1 was relatively complex and difficult to extend, this
> >>> version adopts a simpler anon_vma implementation approach.
> >>
> >> I really hate to repeat myself, but you seemed to have missed my main
> >> point [1]
> >>
> > I saw it, and I’ve waited for a while as well.
> 
> Apparently you saw it but you didn't read my reply about slapping more
> complexity on something complicated.
This v2 is a complete rewrite replacing the original anon_vma and anonymous-page
reverse-mapping traversal.

The new anon_vma hierarchy is represented using a simple doubly linked list with depth information,
replacing the complex topology of red-black trees maintained by each anon_vma in the original design,
and also removing anon_vma_chain.

Excluding patch 11, which optimizes leaf VMA rmap with PVL, and patch 12, which optimizes memory
usage through shared semaphores, the base version contains roughly 800 lines of code, including the
compatibility part replacing the original anon_vma implementation.

> 
>  12 files changed, 1056 insertions(+), 54 deletions(-)
> 
> >
> > If there are technical issues, please point them out directly.
> > Or is this simply how technical discussions are usually conducted in the
> Linux community or the mm subsystem?
> 
> I suggest you go back and read my email.

At first, I thought I was the only one who found anon_vma hard to understand. After reading your earlier email,
I realized that many people also consider it difficult to follow. So I tried a different approach and implemented
anon_vma rmap using a simpler doubly linked list plus depth mechanism.

Thanks,
Tao

> --
> Cheers,
> 
> David

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[Gregory Price]

On Tue, Jul 07, 2026 at 09:20:24AM +0000, wangtao wrote:
> > >
> > > If there are technical issues, please point them out directly.
> > > Or is this simply how technical discussions are usually conducted in the
> > Linux community or the mm subsystem?
> > 
> > I suggest you go back and read my email.
> 
> At first, I thought I was the only one who found anon_vma hard to understand. After reading your earlier email,
> I realized that many people also consider it difficult to follow. So I tried a different approach and implemented
> anon_vma rmap using a simpler doubly linked list plus depth mechanism.
> 

The in-flight was discussed at LSF and then reported on by lwn

https://lwn.net/Articles/1072378/

https://git.kernel.org/pub/scm/linux/kernel/git/ljs/linux.git/log/?h=project/cow-context

If you have issues with the in-flight work and think there's a better
way, it would be good to engage on that front - rather than posting
competing work and expecting maintainers to have to context switch
between two solutions for an objectively hard to understand problem
(which carries significant risk of regressions).

No one is saying you can't engage, just trying to avoid doing the same
work twice for negative value.

~Gregory

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[David Hildenbrand (Arm)]

On 7/7/26 11:20, wangtao wrote:
>>> I saw it, and I’ve waited for a while as well.
>>
>> Apparently you saw it but you didn't read my reply about slapping more
>> complexity on something complicated.
> This v2 is a complete rewrite replacing the original anon_vma and anonymous-page
> reverse-mapping traversal.

Okay, it leaves the old code in place for now. Just as a general node without
going into details: that won't fly. We don't need another MGLRU.

> 
> The new anon_vma hierarchy is represented using a simple doubly linked list with depth information,
> replacing the complex topology of red-black trees maintained by each anon_vma in the original design,
> and also removing anon_vma_chain.

Okay, but anon_vma fundamentally remains. Using a lipstick on a pig.

> 
> Excluding patch 11, which optimizes leaf VMA rmap with PVL, and patch 12, which optimizes memory
> usage through shared semaphores, the base version contains roughly 800 lines of code, including the
> compatibility part replacing the original anon_vma implementation.
> 
>>
>>  12 files changed, 1056 insertions(+), 54 deletions(-)
>>
>>>
>>> If there are technical issues, please point them out directly.
>>> Or is this simply how technical discussions are usually conducted in the
>> Linux community or the mm subsystem?
>>
>> I suggest you go back and read my email.
> 
> At first, I thought I was the only one who found anon_vma hard to understand. After reading your earlier email,
> I realized that many people also consider it difficult to follow. 

Right, that's how Lorenzo started his work, trying to replace/remove anon_vma
entirely.

So I tried a different approach and implemented
> anon_vma rmap using a simpler doubly linked list plus depth mechanism.

And this is what I don't understand. You were told that Lorenzo is working on
removing anon_vma, yet you decided to work on + send something that reworks
anon_vma?

Also, I am really curious about your expectation: not having a single patch
contributed to MM, even worse, not a single RB tag or anything, yet you rework
one of the most complicated part of MM repeatedly using other complicated
approaches, and immediately start demand technical discussions? After sending an
absolute horrible v1?

I really try to be a nice person, but really, am I dreaming?


Obviously, Lorenzo is still working om his approach, about which there were
discussions at conferences and LWN articles.

Stating "that you waited for a while" is ludicrous given that your old patch set
was posted around one month ago, and a complete redesign -- moving away from
anon_vma -- is not expected to be a short-term thing.

Meanwhile, Lorenzo is sending real cleanups [1] as preparation for bigger
changes. ... which is what we usually expect on such fundamental reworks.

I'm really left clueless here why we anyone should spend time trying to thing
through yet another data structure and yet another locking scheme, for something
we want to remove entirely?

Again, I really try to be a nice person, but something here just doesn't add up.

[1] https://lore.kernel.org/r/cover.1782735110.git.ljs@kernel.org

-- 
Cheers,

David

RE: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[wangtao]

> > At first, I thought I was the only one who found anon_vma hard to
> > understand. After reading your earlier email, I realized that many people
> also consider it difficult to follow.
> 
> Right, that's how Lorenzo started his work, trying to replace/remove
> anon_vma entirely.
> 
> So I tried a different approach and implemented
> > anon_vma rmap using a simpler doubly linked list plus depth mechanism.
> 
> And this is what I don't understand. You were told that Lorenzo is working on
> removing anon_vma, yet you decided to work on + send something that
> reworks anon_vma?
> 
> Also, I am really curious about your expectation: not having a single patch
> contributed to MM, even worse, not a single RB tag or anything, yet you
> rework one of the most complicated part of MM repeatedly using other
> complicated approaches, and immediately start demand technical discussions?
> After sending an absolute horrible v1?
> 
> I really try to be a nice person, but really, am I dreaming?
> 

It’s simple: optimize memory usage and share it with the community.

Wish you all the best.

> 
> Obviously, Lorenzo is still working om his approach, about which there were
> discussions at conferences and LWN articles.
> 
> Stating "that you waited for a while" is ludicrous given that your old patch set
> was posted around one month ago, and a complete redesign -- moving away
> from anon_vma -- is not expected to be a short-term thing.
> 
> Meanwhile, Lorenzo is sending real cleanups [1] as preparation for bigger
> changes. ... which is what we usually expect on such fundamental reworks.
> 
> I'm really left clueless here why we anyone should spend time trying to thing
> through yet another data structure and yet another locking scheme, for
> something we want to remove entirely?
> 
> Again, I really try to be a nice person, but something here just doesn't add up.
> 
> [1] https://lore.kernel.org/r/cover.1782735110.git.ljs@kernel.org
> 
> --
> Cheers,
> 
> David

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[David Hildenbrand (Arm)]

On 7/8/26 05:26, wangtao wrote:
>>> At first, I thought I was the only one who found anon_vma hard to
>>> understand. After reading your earlier email, I realized that many people
>> also consider it difficult to follow.
>>
>> Right, that's how Lorenzo started his work, trying to replace/remove
>> anon_vma entirely.
>>
>> So I tried a different approach and implemented
>>> anon_vma rmap using a simpler doubly linked list plus depth mechanism.
>>
>> And this is what I don't understand. You were told that Lorenzo is working on
>> removing anon_vma, yet you decided to work on + send something that
>> reworks anon_vma?
>>
>> Also, I am really curious about your expectation: not having a single patch
>> contributed to MM, even worse, not a single RB tag or anything, yet you
>> rework one of the most complicated part of MM repeatedly using other
>> complicated approaches, and immediately start demand technical discussions?
>> After sending an absolute horrible v1?
>>
>> I really try to be a nice person, but really, am I dreaming?
>>
> 
> It’s simple: optimize memory usage and share it with the community.

Yes, and you can work on whatever you want, and share it with people as much as
you like.

But RFC literally means "Requests For Comments", not "hey, I hacked something
together, maybe this could be useful at some point".

-- 
Cheers,

David

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[Pedro Falcato]

On Tue, Jul 07, 2026 at 08:21:24AM +0000, wangtao wrote:
> > 
> > On 7/7/26 08:32, tao wrote:
> > > Hi all,
> > >
> > > Since v1 was relatively complex and difficult to extend, this version
> > > adopts a simpler anon_vma implementation approach.
> > 
> > I really hate to repeat myself, but you seemed to have missed my main point
> > [1]
> > 
> I saw it, and I’ve waited for a while as well.

what?

A good part of software engineering (and FOSS in general) is collaboration.
People have their own work responsibilities & personal lives. If you had
questions, you could've raised them on-list or directly to Lorenzo and
could've helped him drive it forward (which he is, and he's posting careful
precursor patches).

> 
> If there are technical issues, please point them out directly.
> Or is this simply how technical discussions are usually conducted in the Linux community or the mm subsystem?

There is no technical discussion to be had. Lorenzo is doing a large,
careful rework of anon rmap (and rmap in general). You don't get to override
his work because you decided to ignore the "community outreach and
collaboration" step.

-- 
Pedro

Re: [RFC PATCH v2 00/13] mm: rework anon_vma and remove anon_vma_chain

[Borislav Petkov]

On Tue, Jul 07, 2026 at 02:32:55PM +0800, tao wrote:
>  include/linux/mm.h       |   2 +
>  include/linux/mm_types.h |   2 +
>  include/linux/rmap.h     |  56 ++++
>  include/linux/rwsem.h    |  10 +
>  mm/Kconfig               |  37 +++
>  mm/internal.h            | 205 ++++++++++++
>  mm/ksm.c                 |  19 +-
>  mm/memory-failure.c      |   8 +-
>  mm/mmap.c                |   7 +-
>  mm/rmap.c                | 686 +++++++++++++++++++++++++++++++++++++--
>  mm/vma.c                 |  76 ++++-
>  mm/vma_init.c            |   2 +

Why am I on To: on this diffstat?

Please learn how to use ./scripts/get_maintainer.pl and do not spam unrelated
people with mail.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[syzbot ci] Re: mm: rework anon_vma and remove anon_vma_chain

[syzbot ci]

syzbot ci has tested the following series

[v2] mm: rework anon_vma and remove anon_vma_chain
https://lore.kernel.org/all/20260707063308.29176-1-tao.wangtao@honor.com
* [RFC PATCH v2 01/13] mm: add CONFIG_ANON_VMA_FRACTAL
* [RFC PATCH v2 02/13] mm: implement helpers for ANON_VMA_FRACTAL
* [RFC PATCH v2 03/13] mm: implement __anon_node_prepare for ANON_VMA_FRACTAL
* [RFC PATCH v2 04/13] mm: implement anon_node_clone for ANON_VMA_FRACTAL
* [RFC PATCH v2 05/13] mm: implement anon_node_fork_with_prev for ANON_VMA_FRACTAL
* [RFC PATCH v2 06/13] mm: implement unlink_anon_nodes for ANON_VMA_FRACTAL
* [RFC PATCH v2 07/13] mm: handle rmap_base changes for ANON_VMA_FRACTAL
* [RFC PATCH v2 08/13] mm: implement anonymous folio rmap for ANON_VMA_FRACTAL
* [RFC PATCH v2 09/13] mm: prepare anon_node replacement for ANON_VMA_FRACTAL
* [RFC PATCH v2 10/13] mm: replace anon_vma with anon_node for ANON_VMA_FRACTAL
* [RFC PATCH v2 11/13] mm: optimize rmap for ANON_VMA_FRACTAL with PVL
* [RFC PATCH v2 12/13] mm: shared semaphores for ANON_VMA_FRACTAL
* [RFC PATCH v2 13/13] mm: Enable CONFIG_ANON_VMA_FRACTAL by default

and found the following issue:
WARNING: bad unlock balance in exc_page_fault

Full report is available here:
https://ci.syzbot.org/series/39466cf1-707b-420e-8f1e-11735bea6c66

***

WARNING: bad unlock balance in exc_page_fault

tree:      mm-new
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git
base:      8ff49efd1cf18086320f3b43d1bd483c0b9e7e58
arch:      amd64
compiler:  Debian clang version 22.1.6 (++20260514074242+fc4aad7b5db3-1~exp1~20260514074407.73), Debian LLD 22.1.6
config:    https://ci.syzbot.org/builds/c3a229e4-b949-4889-b309-dbdd1fadd95f/config
syz repro: https://ci.syzbot.org/findings/29b52ad4-b463-41da-877c-4138356e3960/syz_repro

=====================================
WARNING: bad unlock balance detected!
syzkaller #0 Not tainted
-------------------------------------
sshd/5593 is trying to release lock (vm_lock) at:
[<ffffffff8bb5032a>] handle_page_fault arch/x86/mm/fault.c:1483 [inline]
[<ffffffff8bb5032a>] exc_page_fault+0x6a/0xc0 arch/x86/mm/fault.c:1536
but there are no more locks to release!

other info that might help us debug this:
no locks held by sshd/5593.

stack backtrace:
CPU: 1 UID: 0 PID: 5593 Comm: sshd Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
 <TASK>
 dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
 print_unlock_imbalance_bug+0xdc/0xf0 kernel/locking/lockdep.c:5298
 __lock_release kernel/locking/lockdep.c:5527 [inline]
 lock_release+0x1f1/0x3c0 kernel/locking/lockdep.c:5889
 vma_refcount_put include/linux/mmap_lock.h:216 [inline]
 vma_end_read include/linux/mmap_lock.h:264 [inline]
 do_user_addr_fault+0xab0/0x1340 arch/x86/mm/fault.c:1345
 handle_page_fault arch/x86/mm/fault.c:1483 [inline]
 exc_page_fault+0x6a/0xc0 arch/x86/mm/fault.c:1536
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:595
RIP: 0033:0x5583206de68b
Code: 04 25 28 00 00 00 48 89 44 24 08 31 c0 48 89 e2 e8 8c 5b ff ff 85 c0 78 13 31 c0 4d 85 e4 74 0c 48 8b 3c 24 48 89 de 4c 89 e1 <f3> a4 48 8b 54 24 08 64 48 2b 14 25 28 00 00 00 74 05 e8 5e 17 fc
RSP: 002b:00007fff02a37e60 EFLAGS: 00010206
RAX: 0000000000000000 RBX: 000055835a20d6fe RCX: 0000000000008000
RDX: 0000000000000000 RSI: 000055835a20d6fe RDI: 00007fac5106c010
RBP: 000055832073df00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000008000
R13: 000000000000005e R14: 00000000000000b3 R15: 00007fff02a37f88
 </TASK>


***

If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
  Tested-by: syzbot@syzkaller.appspotmail.com

---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.

To test a patch for this bug, please reply with `#syz test`
(should be on a separate line).

The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.