* [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma
2011-02-17 16:19 [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
@ 2011-02-17 16:19 ` Peter Zijlstra
2011-02-17 17:44 ` Rik van Riel
2011-03-09 16:28 ` Mel Gorman
2011-02-17 16:19 ` [PATCH 2/3] mm: Move anon_vma ref out from under CONFIG_foo Peter Zijlstra
` (2 subsequent siblings)
3 siblings, 2 replies; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-17 16:19 UTC (permalink / raw)
To: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, Linus Torvalds
Cc: linux-kernel, linux-mm, Benjamin Herrenschmidt, David Miller,
Hugh Dickins, Mel Gorman, Nick Piggin, Peter Zijlstra,
Paul McKenney, Yanmin Zhang, Hugh Dickins
[-- Attachment #1: peter_zijlstra-mm-rename_drop_anon_vma_to_put_anon_vma.patch --]
[-- Type: text/plain, Size: 4344 bytes --]
The normal code pattern used in the kernel is: get/put.
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/rmap.h | 4 ++--
mm/ksm.c | 23 +++++------------------
mm/migrate.c | 4 ++--
mm/rmap.c | 4 ++--
4 files changed, 11 insertions(+), 24 deletions(-)
Index: linux-2.6/include/linux/rmap.h
===================================================================
--- linux-2.6.orig/include/linux/rmap.h
+++ linux-2.6/include/linux/rmap.h
@@ -87,7 +87,7 @@ static inline void get_anon_vma(struct a
atomic_inc(&anon_vma->external_refcount);
}
-void drop_anon_vma(struct anon_vma *);
+void put_anon_vma(struct anon_vma *);
#else
static inline void anonvma_external_refcount_init(struct anon_vma *anon_vma)
{
@@ -102,7 +102,7 @@ static inline void get_anon_vma(struct a
{
}
-static inline void drop_anon_vma(struct anon_vma *anon_vma)
+static inline void put_anon_vma(struct anon_vma *anon_vma)
{
}
#endif /* CONFIG_KSM */
Index: linux-2.6/mm/ksm.c
===================================================================
--- linux-2.6.orig/mm/ksm.c
+++ linux-2.6/mm/ksm.c
@@ -301,20 +301,6 @@ static inline int in_stable_tree(struct
return rmap_item->address & STABLE_FLAG;
}
-static void hold_anon_vma(struct rmap_item *rmap_item,
- struct anon_vma *anon_vma)
-{
- rmap_item->anon_vma = anon_vma;
- get_anon_vma(anon_vma);
-}
-
-static void ksm_drop_anon_vma(struct rmap_item *rmap_item)
-{
- struct anon_vma *anon_vma = rmap_item->anon_vma;
-
- drop_anon_vma(anon_vma);
-}
-
/*
* ksmd, and unmerge_and_remove_all_rmap_items(), must not touch an mm's
* page tables after it has passed through ksm_exit() - which, if necessary,
@@ -397,7 +383,7 @@ static void break_cow(struct rmap_item *
* It is not an accident that whenever we want to break COW
* to undo, we also need to drop a reference to the anon_vma.
*/
- ksm_drop_anon_vma(rmap_item);
+ put_anon_vma(rmap_item->anon_vma);
down_read(&mm->mmap_sem);
if (ksm_test_exit(mm))
@@ -466,7 +452,7 @@ static void remove_node_from_stable_tree
ksm_pages_sharing--;
else
ksm_pages_shared--;
- ksm_drop_anon_vma(rmap_item);
+ put_anon_vma(rmap_item->anon_vma);
rmap_item->address &= PAGE_MASK;
cond_resched();
}
@@ -554,7 +540,7 @@ static void remove_rmap_item_from_tree(s
else
ksm_pages_shared--;
- ksm_drop_anon_vma(rmap_item);
+ put_anon_vma(rmap_item->anon_vma);
rmap_item->address &= PAGE_MASK;
} else if (rmap_item->address & UNSTABLE_FLAG) {
@@ -949,7 +935,8 @@ static int try_to_merge_with_ksm_page(st
goto out;
/* Must get reference to anon_vma while still holding mmap_sem */
- hold_anon_vma(rmap_item, vma->anon_vma);
+ rmap_item->anon_vma = vma->anon_vma;
+ get_anon_vma(vma->anon_vma);
out:
up_read(&mm->mmap_sem);
return err;
Index: linux-2.6/mm/migrate.c
===================================================================
--- linux-2.6.orig/mm/migrate.c
+++ linux-2.6/mm/migrate.c
@@ -764,7 +764,7 @@ static int unmap_and_move(new_page_t get
/* Drop an anon_vma reference if we took one */
if (anon_vma)
- drop_anon_vma(anon_vma);
+ put_anon_vma(anon_vma);
uncharge:
if (!charge)
@@ -857,7 +857,7 @@ static int unmap_and_move_huge_page(new_
remove_migration_ptes(hpage, hpage);
if (anon_vma)
- drop_anon_vma(anon_vma);
+ put_anon_vma(anon_vma);
out:
unlock_page(hpage);
Index: linux-2.6/mm/rmap.c
===================================================================
--- linux-2.6.orig/mm/rmap.c
+++ linux-2.6/mm/rmap.c
@@ -278,7 +278,7 @@ static void anon_vma_unlink(struct anon_
if (empty) {
/* We no longer need the root anon_vma */
if (anon_vma->root != anon_vma)
- drop_anon_vma(anon_vma->root);
+ put_anon_vma(anon_vma->root);
anon_vma_free(anon_vma);
}
}
@@ -1489,7 +1489,7 @@ int try_to_munlock(struct page *page)
* we know we are the last user, nobody else can get a reference and we
* can do the freeing without the lock.
*/
-void drop_anon_vma(struct anon_vma *anon_vma)
+void put_anon_vma(struct anon_vma *anon_vma)
{
BUG_ON(atomic_read(&anon_vma->external_refcount) <= 0);
if (atomic_dec_and_lock(&anon_vma->external_refcount, &anon_vma->root->lock)) {
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma
2011-02-17 16:19 ` [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma Peter Zijlstra
@ 2011-02-17 17:44 ` Rik van Riel
2011-03-09 16:28 ` Mel Gorman
1 sibling, 0 replies; 14+ messages in thread
From: Rik van Riel @ 2011-02-17 17:44 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Ingo Molnar, akpm,
Linus Torvalds, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On 02/17/2011 11:19 AM, Peter Zijlstra wrote:
Reviewed-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma
2011-02-17 16:19 ` [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma Peter Zijlstra
2011-02-17 17:44 ` Rik van Riel
@ 2011-03-09 16:28 ` Mel Gorman
1 sibling, 0 replies; 14+ messages in thread
From: Mel Gorman @ 2011-03-09 16:28 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, Linus Torvalds, linux-kernel, linux-mm,
Benjamin Herrenschmidt, David Miller, Hugh Dickins, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On Thu, Feb 17, 2011 at 05:19:49PM +0100, Peter Zijlstra wrote:
> The normal code pattern used in the kernel is: get/put.
>
> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> Acked-by: Hugh Dickins <hughd@google.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Mel Gorman <mel@csn.ul.ie>
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 2/3] mm: Move anon_vma ref out from under CONFIG_foo
2011-02-17 16:19 [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
2011-02-17 16:19 ` [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma Peter Zijlstra
@ 2011-02-17 16:19 ` Peter Zijlstra
2011-02-17 17:49 ` Rik van Riel
2011-02-17 16:19 ` [PATCH 3/3] mm: Simplify anon_vma refcounts Peter Zijlstra
2011-02-17 17:36 ` [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
3 siblings, 1 reply; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-17 16:19 UTC (permalink / raw)
To: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, Linus Torvalds
Cc: linux-kernel, linux-mm, Benjamin Herrenschmidt, David Miller,
Hugh Dickins, Mel Gorman, Nick Piggin, Peter Zijlstra,
Paul McKenney, Yanmin Zhang, Hugh Dickins
[-- Attachment #1: peter_zijlstra-mm-move_anon_vma_ref_out_from_under_config_ksm.patch --]
[-- Type: text/plain, Size: 4587 bytes --]
We need the anon_vma refcount unconditionally to simplify the anon_vma
lifetime rules.
Acked-by: Mel Gorman <mel@csn.ul.ie>
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/rmap.h | 40 ++++------------------------------------
mm/rmap.c | 14 ++++++--------
2 files changed, 10 insertions(+), 44 deletions(-)
Index: linux-2.6/include/linux/rmap.h
===================================================================
--- linux-2.6.orig/include/linux/rmap.h
+++ linux-2.6/include/linux/rmap.h
@@ -27,18 +27,15 @@
struct anon_vma {
struct anon_vma *root; /* Root of this anon_vma tree */
spinlock_t lock; /* Serialize access to vma list */
-#if defined(CONFIG_KSM) || defined(CONFIG_MIGRATION)
-
/*
- * The external_refcount is taken by either KSM or page migration
- * to take a reference to an anon_vma when there is no
+ * The refcount is taken on an anon_vma when there is no
* guarantee that the vma of page tables will exist for
* the duration of the operation. A caller that takes
* the reference is responsible for clearing up the
* anon_vma if they are the last user on release
*/
- atomic_t external_refcount;
-#endif
+ atomic_t refcount;
+
/*
* NOTE: the LSB of the head.next is set by
* mm_take_all_locks() _after_ taking the above lock. So the
@@ -71,41 +68,12 @@ struct anon_vma_chain {
};
#ifdef CONFIG_MMU
-#if defined(CONFIG_KSM) || defined(CONFIG_MIGRATION)
-static inline void anonvma_external_refcount_init(struct anon_vma *anon_vma)
-{
- atomic_set(&anon_vma->external_refcount, 0);
-}
-
-static inline int anonvma_external_refcount(struct anon_vma *anon_vma)
-{
- return atomic_read(&anon_vma->external_refcount);
-}
-
static inline void get_anon_vma(struct anon_vma *anon_vma)
{
- atomic_inc(&anon_vma->external_refcount);
+ atomic_inc(&anon_vma->refcount);
}
void put_anon_vma(struct anon_vma *);
-#else
-static inline void anonvma_external_refcount_init(struct anon_vma *anon_vma)
-{
-}
-
-static inline int anonvma_external_refcount(struct anon_vma *anon_vma)
-{
- return 0;
-}
-
-static inline void get_anon_vma(struct anon_vma *anon_vma)
-{
-}
-
-static inline void put_anon_vma(struct anon_vma *anon_vma)
-{
-}
-#endif /* CONFIG_KSM */
static inline struct anon_vma *page_anon_vma(struct page *page)
{
Index: linux-2.6/mm/rmap.c
===================================================================
--- linux-2.6.orig/mm/rmap.c
+++ linux-2.6/mm/rmap.c
@@ -272,7 +272,7 @@ static void anon_vma_unlink(struct anon_
list_del(&anon_vma_chain->same_anon_vma);
/* We must garbage collect the anon_vma if it's empty */
- empty = list_empty(&anon_vma->head) && !anonvma_external_refcount(anon_vma);
+ empty = list_empty(&anon_vma->head) && !atomic_read(&anon_vma->refcount);
anon_vma_unlock(anon_vma);
if (empty) {
@@ -303,7 +303,7 @@ static void anon_vma_ctor(void *data)
struct anon_vma *anon_vma = data;
spin_lock_init(&anon_vma->lock);
- anonvma_external_refcount_init(anon_vma);
+ atomic_set(&anon_vma->refcount, 0);
INIT_LIST_HEAD(&anon_vma->head);
}
@@ -1482,7 +1482,6 @@ int try_to_munlock(struct page *page)
return try_to_unmap_file(page, TTU_MUNLOCK);
}
-#if defined(CONFIG_KSM) || defined(CONFIG_MIGRATION)
/*
* Drop an anon_vma refcount, freeing the anon_vma and anon_vma->root
* if necessary. Be careful to do all the tests under the lock. Once
@@ -1491,8 +1490,8 @@ int try_to_munlock(struct page *page)
*/
void put_anon_vma(struct anon_vma *anon_vma)
{
- BUG_ON(atomic_read(&anon_vma->external_refcount) <= 0);
- if (atomic_dec_and_lock(&anon_vma->external_refcount, &anon_vma->root->lock)) {
+ BUG_ON(atomic_read(&anon_vma->refcount) <= 0);
+ if (atomic_dec_and_lock(&anon_vma->refcount, &anon_vma->root->lock)) {
struct anon_vma *root = anon_vma->root;
int empty = list_empty(&anon_vma->head);
int last_root_user = 0;
@@ -1503,8 +1502,8 @@ void put_anon_vma(struct anon_vma *anon_
* the refcount on the root and check if we need to free it.
*/
if (empty && anon_vma != root) {
- BUG_ON(atomic_read(&root->external_refcount) <= 0);
- last_root_user = atomic_dec_and_test(&root->external_refcount);
+ BUG_ON(atomic_read(&root->refcount) <= 0);
+ last_root_user = atomic_dec_and_test(&root->refcount);
root_empty = list_empty(&root->head);
}
anon_vma_unlock(anon_vma);
@@ -1516,7 +1515,6 @@ void put_anon_vma(struct anon_vma *anon_
}
}
}
-#endif
#ifdef CONFIG_MIGRATION
/*
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 2/3] mm: Move anon_vma ref out from under CONFIG_foo
2011-02-17 16:19 ` [PATCH 2/3] mm: Move anon_vma ref out from under CONFIG_foo Peter Zijlstra
@ 2011-02-17 17:49 ` Rik van Riel
0 siblings, 0 replies; 14+ messages in thread
From: Rik van Riel @ 2011-02-17 17:49 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Ingo Molnar, akpm,
Linus Torvalds, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On 02/17/2011 11:19 AM, Peter Zijlstra wrote:
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-17 16:19 [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
2011-02-17 16:19 ` [PATCH 1/3] mm: Rename drop_anon_vma to put_anon_vma Peter Zijlstra
2011-02-17 16:19 ` [PATCH 2/3] mm: Move anon_vma ref out from under CONFIG_foo Peter Zijlstra
@ 2011-02-17 16:19 ` Peter Zijlstra
2011-02-17 17:47 ` Rik van Riel
2011-02-17 18:30 ` Linus Torvalds
2011-02-17 17:36 ` [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
3 siblings, 2 replies; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-17 16:19 UTC (permalink / raw)
To: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, Linus Torvalds
Cc: linux-kernel, linux-mm, Benjamin Herrenschmidt, David Miller,
Hugh Dickins, Mel Gorman, Nick Piggin, Peter Zijlstra,
Paul McKenney, Yanmin Zhang, Hugh Dickins
[-- Attachment #1: peter_zijlstra-mm-simplify_anon_vma_refcounts.patch --]
[-- Type: text/plain, Size: 6088 bytes --]
This patch changes the anon_vma refcount to be 0 when the object is
free. It does this by adding 1 ref to being in use in the anon_vma
structure (iow. the anon_vma->head list is not empty).
This allows a simpler release scheme without having to check both the
refcount and the list as well as avoids taking a ref for each entry
on the list.
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/rmap.h | 11 +++++--
mm/rmap.c | 79 ++++++++++++++++++---------------------------------
2 files changed, 37 insertions(+), 53 deletions(-)
Index: linux-2.6/include/linux/rmap.h
===================================================================
--- linux-2.6.orig/include/linux/rmap.h
+++ linux-2.6/include/linux/rmap.h
@@ -73,7 +73,13 @@ static inline void get_anon_vma(struct a
atomic_inc(&anon_vma->refcount);
}
-void put_anon_vma(struct anon_vma *);
+void __put_anon_vma(struct anon_vma *anon_vma);
+
+static inline void put_anon_vma(struct anon_vma *anon_vma)
+{
+ if (atomic_dec_and_test(&anon_vma->refcount))
+ __put_anon_vma(anon_vma);
+}
static inline struct anon_vma *page_anon_vma(struct page *page)
{
@@ -116,7 +122,6 @@ void unlink_anon_vmas(struct vm_area_str
int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *);
int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *);
void __anon_vma_link(struct vm_area_struct *);
-void anon_vma_free(struct anon_vma *);
static inline void anon_vma_merge(struct vm_area_struct *vma,
struct vm_area_struct *next)
@@ -125,6 +130,8 @@ static inline void anon_vma_merge(struct
unlink_anon_vmas(next);
}
+struct anon_vma *page_get_anon_vma(struct page *page);
+
/*
* rmap interfaces called when adding or removing pte of page
*/
Index: linux-2.6/mm/rmap.c
===================================================================
--- linux-2.6.orig/mm/rmap.c
+++ linux-2.6/mm/rmap.c
@@ -67,11 +67,24 @@ static struct kmem_cache *anon_vma_chain
static inline struct anon_vma *anon_vma_alloc(void)
{
- return kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL);
+ struct anon_vma *anon_vma;
+
+ anon_vma = kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL);
+ if (anon_vma) {
+ atomic_set(&anon_vma->refcount, 1);
+ /*
+ * Initialise the anon_vma root to point to itself. If called
+ * from fork, the root will be reset to the parents anon_vma.
+ */
+ anon_vma->root = anon_vma;
+ }
+
+ return anon_vma;
}
-void anon_vma_free(struct anon_vma *anon_vma)
+static inline void anon_vma_free(struct anon_vma *anon_vma)
{
+ VM_BUG_ON(atomic_read(&anon_vma->refcount));
kmem_cache_free(anon_vma_cachep, anon_vma);
}
@@ -133,11 +146,6 @@ int anon_vma_prepare(struct vm_area_stru
if (unlikely(!anon_vma))
goto out_enomem_free_avc;
allocated = anon_vma;
- /*
- * This VMA had no anon_vma yet. This anon_vma is
- * the root of any anon_vma tree that might form.
- */
- anon_vma->root = anon_vma;
}
anon_vma_lock(anon_vma);
@@ -156,7 +164,7 @@ int anon_vma_prepare(struct vm_area_stru
anon_vma_unlock(anon_vma);
if (unlikely(allocated))
- anon_vma_free(allocated);
+ put_anon_vma(allocated);
if (unlikely(avc))
anon_vma_chain_free(avc);
}
@@ -241,9 +249,9 @@ int anon_vma_fork(struct vm_area_struct
*/
anon_vma->root = pvma->anon_vma->root;
/*
- * With KSM refcounts, an anon_vma can stay around longer than the
- * process it belongs to. The root anon_vma needs to be pinned
- * until this anon_vma is freed, because the lock lives in the root.
+ * With refcounts, an anon_vma can stay around longer than the
+ * process it belongs to. The root anon_vma needs to be pinned until
+ * this anon_vma is freed, because the lock lives in the root.
*/
get_anon_vma(anon_vma->root);
/* Mark this anon_vma as the one where our new (COWed) pages go. */
@@ -253,7 +261,7 @@ int anon_vma_fork(struct vm_area_struct
return 0;
out_error_free_anon_vma:
- anon_vma_free(anon_vma);
+ put_anon_vma(anon_vma);
out_error:
unlink_anon_vmas(vma);
return -ENOMEM;
@@ -272,15 +280,11 @@ static void anon_vma_unlink(struct anon_
list_del(&anon_vma_chain->same_anon_vma);
/* We must garbage collect the anon_vma if it's empty */
- empty = list_empty(&anon_vma->head) && !atomic_read(&anon_vma->refcount);
+ empty = list_empty(&anon_vma->head);
anon_vma_unlock(anon_vma);
- if (empty) {
- /* We no longer need the root anon_vma */
- if (anon_vma->root != anon_vma)
- put_anon_vma(anon_vma->root);
- anon_vma_free(anon_vma);
- }
+ if (empty)
+ put_anon_vma(anon_vma);
}
void unlink_anon_vmas(struct vm_area_struct *vma)
@@ -1482,38 +1486,11 @@ int try_to_munlock(struct page *page)
return try_to_unmap_file(page, TTU_MUNLOCK);
}
-/*
- * Drop an anon_vma refcount, freeing the anon_vma and anon_vma->root
- * if necessary. Be careful to do all the tests under the lock. Once
- * we know we are the last user, nobody else can get a reference and we
- * can do the freeing without the lock.
- */
-void put_anon_vma(struct anon_vma *anon_vma)
-{
- BUG_ON(atomic_read(&anon_vma->refcount) <= 0);
- if (atomic_dec_and_lock(&anon_vma->refcount, &anon_vma->root->lock)) {
- struct anon_vma *root = anon_vma->root;
- int empty = list_empty(&anon_vma->head);
- int last_root_user = 0;
- int root_empty = 0;
-
- /*
- * The refcount on a non-root anon_vma got dropped. Drop
- * the refcount on the root and check if we need to free it.
- */
- if (empty && anon_vma != root) {
- BUG_ON(atomic_read(&root->refcount) <= 0);
- last_root_user = atomic_dec_and_test(&root->refcount);
- root_empty = list_empty(&root->head);
- }
- anon_vma_unlock(anon_vma);
-
- if (empty) {
- anon_vma_free(anon_vma);
- if (root_empty && last_root_user)
- anon_vma_free(root);
- }
- }
+void __put_anon_vma(struct anon_vma *anon_vma)
+{
+ if (anon_vma->root != anon_vma)
+ put_anon_vma(anon_vma->root);
+ anon_vma_free(anon_vma);
}
#ifdef CONFIG_MIGRATION
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-17 16:19 ` [PATCH 3/3] mm: Simplify anon_vma refcounts Peter Zijlstra
@ 2011-02-17 17:47 ` Rik van Riel
2011-02-17 18:30 ` Linus Torvalds
1 sibling, 0 replies; 14+ messages in thread
From: Rik van Riel @ 2011-02-17 17:47 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Ingo Molnar, akpm,
Linus Torvalds, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On 02/17/2011 11:19 AM, Peter Zijlstra wrote:
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-17 16:19 ` [PATCH 3/3] mm: Simplify anon_vma refcounts Peter Zijlstra
2011-02-17 17:47 ` Rik van Riel
@ 2011-02-17 18:30 ` Linus Torvalds
2011-02-18 11:30 ` Peter Zijlstra
2011-02-18 13:44 ` Peter Zijlstra
1 sibling, 2 replies; 14+ messages in thread
From: Linus Torvalds @ 2011-02-17 18:30 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On Thu, Feb 17, 2011 at 8:19 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>
> +void __put_anon_vma(struct anon_vma *anon_vma)
> +{
> + if (anon_vma->root != anon_vma)
> + put_anon_vma(anon_vma->root);
> + anon_vma_free(anon_vma);
> }
So this makes me nervous. It looks like recursion.
Now, I don't think we can ever get a chain of these things (because
the root should be the root of everything), but I still preferred the
older code that made that "one-level root" case explicit, and didn't
have recursion.
IOW, even though it should be entirely equivalent, I think I'd really
prefer something like
void __put_anon_vma(struct anon_vma *anon_vma)
{
struct anon_vma *root = anon_vma->root;
if (root != anon_vma && atomic_dec_and_test(&root->refcount))
anon_vma_free(root);
anon_vma_free(anon_vma);
}
instead. Exactly because it makes it very clear that the "root" is a
root, and we're not doing some possibly arbitrarily deep list like the
dentry tree (which avoids recursion by open-coding its freeing as a
loop).
Hmm? (The above is obviously untested, maybe it has some stupid bug)
Linus
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-17 18:30 ` Linus Torvalds
@ 2011-02-18 11:30 ` Peter Zijlstra
2011-02-18 13:44 ` Peter Zijlstra
1 sibling, 0 replies; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-18 11:30 UTC (permalink / raw)
To: Linus Torvalds
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On Thu, 2011-02-17 at 10:30 -0800, Linus Torvalds wrote:
> On Thu, Feb 17, 2011 at 8:19 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> >
> > +void __put_anon_vma(struct anon_vma *anon_vma)
> > +{
> > + if (anon_vma->root != anon_vma)
> > + put_anon_vma(anon_vma->root);
> > + anon_vma_free(anon_vma);
> > }
>
> So this makes me nervous. It looks like recursion.
>
> Now, I don't think we can ever get a chain of these things (because
> the root should be the root of everything),
Exactly.
> but I still preferred the
> older code that made that "one-level root" case explicit, and didn't
> have recursion.
>
> IOW, even though it should be entirely equivalent, I think I'd really
> prefer something like
>
> void __put_anon_vma(struct anon_vma *anon_vma)
> {
> struct anon_vma *root = anon_vma->root;
>
> if (root != anon_vma && atomic_dec_and_test(&root->refcount))
> anon_vma_free(root);
> anon_vma_free(anon_vma);
> }
>
> instead. Exactly because it makes it very clear that the "root" is a
> root, and we're not doing some possibly arbitrarily deep list like the
> dentry tree (which avoids recursion by open-coding its freeing as a
> loop).
>
> Hmm? (The above is obviously untested, maybe it has some stupid bug)
Looks about right, I'll give it a spin.
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-17 18:30 ` Linus Torvalds
2011-02-18 11:30 ` Peter Zijlstra
@ 2011-02-18 13:44 ` Peter Zijlstra
2011-02-18 14:49 ` Rik van Riel
2011-03-09 16:38 ` Mel Gorman
1 sibling, 2 replies; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-18 13:44 UTC (permalink / raw)
To: Linus Torvalds
Cc: Andrea Arcangeli, Avi Kivity, Thomas Gleixner, Rik van Riel,
Ingo Molnar, akpm, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
Subject: mm: Simplify anon_vma refcounts
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Fri, 26 Nov 2010 15:38:49 +0100
This patch changes the anon_vma refcount to be 0 when the object is
free. It does this by adding 1 ref to being in use in the anon_vma
structure (iow. the anon_vma->head list is not empty).
This allows a simpler release scheme without having to check both the
refcount and the list as well as avoids taking a ref for each entry
on the list.
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/rmap.h | 11 +++++--
mm/rmap.c | 78 ++++++++++++++++++---------------------------------
2 files changed, 38 insertions(+), 51 deletions(-)
Index: linux-2.6/include/linux/rmap.h
===================================================================
--- linux-2.6.orig/include/linux/rmap.h
+++ linux-2.6/include/linux/rmap.h
@@ -73,7 +73,13 @@ static inline void get_anon_vma(struct a
atomic_inc(&anon_vma->refcount);
}
-void put_anon_vma(struct anon_vma *);
+void __put_anon_vma(struct anon_vma *anon_vma);
+
+static inline void put_anon_vma(struct anon_vma *anon_vma)
+{
+ if (atomic_dec_and_test(&anon_vma->refcount))
+ __put_anon_vma(anon_vma);
+}
static inline struct anon_vma *page_anon_vma(struct page *page)
{
@@ -116,7 +122,6 @@ void unlink_anon_vmas(struct vm_area_str
int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *);
int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *);
void __anon_vma_link(struct vm_area_struct *);
-void anon_vma_free(struct anon_vma *);
static inline void anon_vma_merge(struct vm_area_struct *vma,
struct vm_area_struct *next)
@@ -125,6 +130,8 @@ static inline void anon_vma_merge(struct
unlink_anon_vmas(next);
}
+struct anon_vma *page_get_anon_vma(struct page *page);
+
/*
* rmap interfaces called when adding or removing pte of page
*/
Index: linux-2.6/mm/rmap.c
===================================================================
--- linux-2.6.orig/mm/rmap.c
+++ linux-2.6/mm/rmap.c
@@ -67,11 +67,24 @@ static struct kmem_cache *anon_vma_chain
static inline struct anon_vma *anon_vma_alloc(void)
{
- return kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL);
+ struct anon_vma *anon_vma;
+
+ anon_vma = kmem_cache_alloc(anon_vma_cachep, GFP_KERNEL);
+ if (anon_vma) {
+ atomic_set(&anon_vma->refcount, 1);
+ /*
+ * Initialise the anon_vma root to point to itself. If called
+ * from fork, the root will be reset to the parents anon_vma.
+ */
+ anon_vma->root = anon_vma;
+ }
+
+ return anon_vma;
}
-void anon_vma_free(struct anon_vma *anon_vma)
+static inline void anon_vma_free(struct anon_vma *anon_vma)
{
+ VM_BUG_ON(atomic_read(&anon_vma->refcount));
kmem_cache_free(anon_vma_cachep, anon_vma);
}
@@ -133,11 +146,6 @@ int anon_vma_prepare(struct vm_area_stru
if (unlikely(!anon_vma))
goto out_enomem_free_avc;
allocated = anon_vma;
- /*
- * This VMA had no anon_vma yet. This anon_vma is
- * the root of any anon_vma tree that might form.
- */
- anon_vma->root = anon_vma;
}
anon_vma_lock(anon_vma);
@@ -156,7 +164,7 @@ int anon_vma_prepare(struct vm_area_stru
anon_vma_unlock(anon_vma);
if (unlikely(allocated))
- anon_vma_free(allocated);
+ put_anon_vma(allocated);
if (unlikely(avc))
anon_vma_chain_free(avc);
}
@@ -241,9 +249,9 @@ int anon_vma_fork(struct vm_area_struct
*/
anon_vma->root = pvma->anon_vma->root;
/*
- * With KSM refcounts, an anon_vma can stay around longer than the
- * process it belongs to. The root anon_vma needs to be pinned
- * until this anon_vma is freed, because the lock lives in the root.
+ * With refcounts, an anon_vma can stay around longer than the
+ * process it belongs to. The root anon_vma needs to be pinned until
+ * this anon_vma is freed, because the lock lives in the root.
*/
get_anon_vma(anon_vma->root);
/* Mark this anon_vma as the one where our new (COWed) pages go. */
@@ -253,7 +261,7 @@ int anon_vma_fork(struct vm_area_struct
return 0;
out_error_free_anon_vma:
- anon_vma_free(anon_vma);
+ put_anon_vma(anon_vma);
out_error:
unlink_anon_vmas(vma);
return -ENOMEM;
@@ -272,15 +280,11 @@ static void anon_vma_unlink(struct anon_
list_del(&anon_vma_chain->same_anon_vma);
/* We must garbage collect the anon_vma if it's empty */
- empty = list_empty(&anon_vma->head) && !atomic_read(&anon_vma->refcount);
+ empty = list_empty(&anon_vma->head);
anon_vma_unlock(anon_vma);
- if (empty) {
- /* We no longer need the root anon_vma */
- if (anon_vma->root != anon_vma)
- put_anon_vma(anon_vma->root);
- anon_vma_free(anon_vma);
- }
+ if (empty)
+ put_anon_vma(anon_vma);
}
void unlink_anon_vmas(struct vm_area_struct *vma)
@@ -1470,38 +1474,14 @@ int try_to_munlock(struct page *page)
return try_to_unmap_file(page, TTU_MUNLOCK);
}
-/*
- * Drop an anon_vma refcount, freeing the anon_vma and anon_vma->root
- * if necessary. Be careful to do all the tests under the lock. Once
- * we know we are the last user, nobody else can get a reference and we
- * can do the freeing without the lock.
- */
-void put_anon_vma(struct anon_vma *anon_vma)
-{
- BUG_ON(atomic_read(&anon_vma->refcount) <= 0);
- if (atomic_dec_and_lock(&anon_vma->refcount, &anon_vma->root->lock)) {
- struct anon_vma *root = anon_vma->root;
- int empty = list_empty(&anon_vma->head);
- int last_root_user = 0;
- int root_empty = 0;
+void __put_anon_vma(struct anon_vma *anon_vma)
+{
+ struct anon_vma *root = anon_vma->root;
- /*
- * The refcount on a non-root anon_vma got dropped. Drop
- * the refcount on the root and check if we need to free it.
- */
- if (empty && anon_vma != root) {
- BUG_ON(atomic_read(&root->refcount) <= 0);
- last_root_user = atomic_dec_and_test(&root->refcount);
- root_empty = list_empty(&root->head);
- }
- anon_vma_unlock(anon_vma);
+ if (root != anon_vma && atomic_dec_and_test(&root->refcount))
+ anon_vma_free(root);
- if (empty) {
- anon_vma_free(anon_vma);
- if (root_empty && last_root_user)
- anon_vma_free(root);
- }
- }
+ anon_vma_free(anon_vma);
}
#ifdef CONFIG_MIGRATION
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-18 13:44 ` Peter Zijlstra
@ 2011-02-18 14:49 ` Rik van Riel
2011-03-09 16:38 ` Mel Gorman
1 sibling, 0 replies; 14+ messages in thread
From: Rik van Riel @ 2011-02-18 14:49 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Linus Torvalds, Andrea Arcangeli, Avi Kivity, Thomas Gleixner,
Ingo Molnar, akpm, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On 02/18/2011 08:44 AM, Peter Zijlstra wrote:
> Subject: mm: Simplify anon_vma refcounts
> From: Peter Zijlstra<a.p.zijlstra@chello.nl>
> Date: Fri, 26 Nov 2010 15:38:49 +0100
>
> This patch changes the anon_vma refcount to be 0 when the object is
> free. It does this by adding 1 ref to being in use in the anon_vma
> structure (iow. the anon_vma->head list is not empty).
>
> This allows a simpler release scheme without having to check both the
> refcount and the list as well as avoids taking a ref for each entry
> on the list.
>
> Reviewed-by: KAMEZAWA Hiroyuki<kamezawa.hiroyu@jp.fujitsu.com>
> Acked-by: Hugh Dickins<hughd@google.com>
> Signed-off-by: Peter Zijlstra<a.p.zijlstra@chello.nl>
Acked-by: Rik van Riel <riel@redhat.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] mm: Simplify anon_vma refcounts
2011-02-18 13:44 ` Peter Zijlstra
2011-02-18 14:49 ` Rik van Riel
@ 2011-03-09 16:38 ` Mel Gorman
1 sibling, 0 replies; 14+ messages in thread
From: Mel Gorman @ 2011-03-09 16:38 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Linus Torvalds, Andrea Arcangeli, Avi Kivity, Thomas Gleixner,
Rik van Riel, Ingo Molnar, akpm, linux-kernel, linux-mm,
Benjamin Herrenschmidt, David Miller, Hugh Dickins, Nick Piggin,
Paul McKenney, Yanmin Zhang, Hugh Dickins
On Fri, Feb 18, 2011 at 02:44:43PM +0100, Peter Zijlstra wrote:
> Subject: mm: Simplify anon_vma refcounts
> From: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Fri, 26 Nov 2010 15:38:49 +0100
>
> This patch changes the anon_vma refcount to be 0 when the object is
> free. It does this by adding 1 ref to being in use in the anon_vma
> structure (iow. the anon_vma->head list is not empty).
>
> This allows a simpler release scheme without having to check both the
> refcount and the list as well as avoids taking a ref for each entry
> on the list.
>
> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> Acked-by: Hugh Dickins <hughd@google.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Mel Gorman <mel@csn.ul.ie>
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/3] mm: Simplify anon_vma lifetime rules
2011-02-17 16:19 [PATCH 0/3] mm: Simplify anon_vma lifetime rules Peter Zijlstra
` (2 preceding siblings ...)
2011-02-17 16:19 ` [PATCH 3/3] mm: Simplify anon_vma refcounts Peter Zijlstra
@ 2011-02-17 17:36 ` Peter Zijlstra
3 siblings, 0 replies; 14+ messages in thread
From: Peter Zijlstra @ 2011-02-17 17:36 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Avi Kivity, Thomas Gleixner, Rik van Riel, Ingo Molnar, akpm,
Linus Torvalds, linux-kernel, linux-mm, Benjamin Herrenschmidt,
David Miller, Hugh Dickins, Mel Gorman, Nick Piggin,
Paul McKenney, Yanmin Zhang
On Thu, 2011-02-17 at 17:19 +0100, Peter Zijlstra wrote:
> As per Linus' request, isolate these three patches.
---
include/linux/rmap.h | 45 ++++++---------------------
mm/ksm.c | 23 +++-----------
mm/migrate.c | 4 +-
mm/rmap.c | 83 +++++++++++++++++----------------------------------
4 files changed, 46 insertions(+), 109 deletions(-)
^ permalink raw reply [flat|nested] 14+ messages in thread