From: Ingo Molnar <mingo@kernel.org>
To: George Spelvin <linux@horizon.com>
Cc: dave@sr71.net, linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux@rasmusvillemoes.dk, peterz@infradead.org, riel@redhat.com,
rientjes@google.com, torvalds@linux-foundation.org
Subject: [PATCH 3/3 v5] mm/vmalloc: Cache the vmalloc memory info
Date: Mon, 24 Aug 2015 09:50:18 +0200 [thread overview]
Message-ID: <20150824075018.GB20106@gmail.com> (raw)
In-Reply-To: <20150824074714.GA20106@gmail.com>
* Ingo Molnar <mingo@kernel.org> wrote:
> One more detail: I just realized that with the read barriers, the READ_ONCE()
> accesses are not needed anymore - the barriers and the control dependencies are
> enough.
>
> This will further simplify the code.
I.e. something like the updated patch below. (We still need the WRITE_ONCE() for
vmap_info_gen update.)
Thanks,
Ingo
========================>
WARNING: multiple messages have this Message-ID (diff)
From: Ingo Molnar <mingo@kernel.org>
To: George Spelvin <linux@horizon.com>
Cc: dave@sr71.net, linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux@rasmusvillemoes.dk, peterz@infradead.org, riel@redhat.com,
rientjes@google.com, torvalds@linux-foundation.org
Subject: [PATCH 3/3 v5] mm/vmalloc: Cache the vmalloc memory info
Date: Mon, 24 Aug 2015 09:50:18 +0200 [thread overview]
Message-ID: <20150824075018.GB20106@gmail.com> (raw)
In-Reply-To: <20150824074714.GA20106@gmail.com>
* Ingo Molnar <mingo@kernel.org> wrote:
> One more detail: I just realized that with the read barriers, the READ_ONCE()
> accesses are not needed anymore - the barriers and the control dependencies are
> enough.
>
> This will further simplify the code.
I.e. something like the updated patch below. (We still need the WRITE_ONCE() for
vmap_info_gen update.)
Thanks,
Ingo
========================>
>From 46a0507e0a395a7bc2fe4b46a4766e7457ac0140 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@kernel.org>
Date: Sat, 22 Aug 2015 12:28:01 +0200
Subject: [PATCH] mm/vmalloc: Cache the vmalloc memory info
Linus reported that for scripting-intense workloads such as the
Git build, glibc's qsort will read /proc/meminfo for every process
created (by way of get_phys_pages()), which causes the Git build
to generate a surprising amount of kernel overhead.
A fair chunk of the overhead is due to get_vmalloc_info() - which
walks a potentially long list to do its statistics.
Modify Linus's jiffies based patch to use generation counters
to cache the vmalloc info: vmap_unlock() increases the generation
counter, and the get_vmalloc_info() reads it and compares it
against a cached generation counter.
Also use a seqlock to make sure we always print a consistent
set of vmalloc statistics.
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: linux-mm@kvack.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
mm/vmalloc.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 79 insertions(+), 3 deletions(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 605138083880..2f8d9660e007 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -276,7 +276,21 @@ EXPORT_SYMBOL(vmalloc_to_pfn);
#define VM_LAZY_FREEING 0x02
#define VM_VM_AREA 0x04
-static DEFINE_SPINLOCK(vmap_area_lock);
+static __cacheline_aligned_in_smp DEFINE_SPINLOCK(vmap_area_lock);
+
+#ifdef CONFIG_PROC_FS
+/*
+ * A seqlock and two generation counters for a simple cache of the
+ * vmalloc allocation statistics info printed in /proc/meminfo.
+ *
+ * ( The assumption of the optimization is that it's read frequently, but
+ * modified infrequently. )
+ */
+static DEFINE_SPINLOCK(vmap_info_lock);
+static int vmap_info_gen = 1;
+static int vmap_info_cache_gen;
+static struct vmalloc_info vmap_info_cache;
+#endif
static inline void vmap_lock(void)
{
@@ -285,6 +299,9 @@ static inline void vmap_lock(void)
static inline void vmap_unlock(void)
{
+#ifdef CONFIG_PROC_FS
+ WRITE_ONCE(vmap_info_gen, vmap_info_gen+1);
+#endif
spin_unlock(&vmap_area_lock);
}
@@ -2699,7 +2716,7 @@ static int __init proc_vmalloc_init(void)
}
module_init(proc_vmalloc_init);
-void get_vmalloc_info(struct vmalloc_info *vmi)
+static void calc_vmalloc_info(struct vmalloc_info *vmi)
{
struct vmap_area *va;
unsigned long free_area_size;
@@ -2746,5 +2763,64 @@ void get_vmalloc_info(struct vmalloc_info *vmi)
out:
rcu_read_unlock();
}
-#endif
+/*
+ * Return a consistent snapshot of the current vmalloc allocation
+ * statistics, for /proc/meminfo:
+ */
+void get_vmalloc_info(struct vmalloc_info *vmi)
+{
+ int gen = vmap_info_gen;
+
+ /*
+ * If the generation counter of the cache matches that of
+ * the vmalloc generation counter then return the cache:
+ */
+ if (vmap_info_cache_gen == gen) {
+ int gen_after;
+
+ /*
+ * The two read barriers make sure that we read
+ * 'gen', 'vmap_info_cache' and 'gen_after' in
+ * precisely that order:
+ */
+ smp_rmb();
+ *vmi = vmap_info_cache;
+
+ smp_rmb();
+ gen_after = vmap_info_gen;
+
+ /* The cache is still valid: */
+ if (gen == gen_after)
+ return;
+
+ /* Ok, the cache got invalidated just now, regenerate it */
+ gen = gen_after;
+ }
+
+ /* Make sure 'gen' is read before the vmalloc info */
+ smp_rmb();
+
+ calc_vmalloc_info(vmi);
+
+ /*
+ * All updates to vmap_info_cache_gen go through this spinlock,
+ * so when the cache got invalidated, we'll only mark it valid
+ * again if we first fully write the new vmap_info_cache.
+ *
+ * This ensures that partial results won't be used.
+ */
+ spin_lock(&vmap_info_lock);
+ if (gen-vmap_info_cache_gen > 0) {
+ vmap_info_cache = *vmi;
+ /*
+ * Make sure the new cached data is visible before
+ * the generation counter update:
+ */
+ smp_wmb();
+ vmap_info_cache_gen = gen;
+ }
+ spin_unlock(&vmap_info_lock);
+}
+
+#endif /* CONFIG_PROC_FS */
next prev parent reply other threads:[~2015-08-24 7:50 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-08-23 4:48 [PATCH 0/3] mm/vmalloc: Cache the /proc/meminfo vmalloc statistics George Spelvin
2015-08-23 4:48 ` George Spelvin
2015-08-23 6:04 ` Ingo Molnar
2015-08-23 6:04 ` Ingo Molnar
2015-08-23 6:46 ` George Spelvin
2015-08-23 6:46 ` George Spelvin
2015-08-23 8:17 ` [PATCH 3/3 v3] mm/vmalloc: Cache the vmalloc memory info Ingo Molnar
2015-08-23 8:17 ` Ingo Molnar
2015-08-23 20:53 ` Rasmus Villemoes
2015-08-23 20:53 ` Rasmus Villemoes
2015-08-24 6:58 ` Ingo Molnar
2015-08-24 6:58 ` Ingo Molnar
2015-08-24 8:39 ` Rasmus Villemoes
2015-08-24 8:39 ` Rasmus Villemoes
2015-08-23 21:56 ` Rasmus Villemoes
2015-08-23 21:56 ` Rasmus Villemoes
2015-08-24 7:00 ` Ingo Molnar
2015-08-24 7:00 ` Ingo Molnar
2015-08-25 16:39 ` Linus Torvalds
2015-08-25 16:39 ` Linus Torvalds
2015-08-25 17:03 ` Linus Torvalds
2015-08-25 17:03 ` Linus Torvalds
2015-08-24 1:04 ` George Spelvin
2015-08-24 1:04 ` George Spelvin
2015-08-24 7:34 ` [PATCH 3/3 v4] " Ingo Molnar
2015-08-24 7:34 ` Ingo Molnar
2015-08-24 7:47 ` Ingo Molnar
2015-08-24 7:47 ` Ingo Molnar
2015-08-24 7:50 ` Ingo Molnar [this message]
2015-08-24 7:50 ` [PATCH 3/3 v5] " Ingo Molnar
2015-08-24 12:54 ` George Spelvin
2015-08-24 12:54 ` George Spelvin
2015-08-25 9:56 ` [PATCH 3/3 v6] " Ingo Molnar
2015-08-25 9:56 ` Ingo Molnar
2015-08-25 10:36 ` George Spelvin
2015-08-25 10:36 ` George Spelvin
2015-08-25 12:59 ` Peter Zijlstra
2015-08-25 12:59 ` Peter Zijlstra
2015-08-25 14:19 ` Rasmus Villemoes
2015-08-25 14:19 ` Rasmus Villemoes
2015-08-25 15:11 ` George Spelvin
2015-08-25 15:11 ` George Spelvin
2015-08-24 13:11 ` [PATCH 3/3 v4] " John Stoffel
2015-08-24 13:11 ` John Stoffel
2015-08-24 15:11 ` George Spelvin
2015-08-24 15:11 ` George Spelvin
2015-08-24 15:55 ` John Stoffel
2015-08-24 15:55 ` John Stoffel
2015-08-25 12:46 ` [PATCH 3/3 v3] " Peter Zijlstra
2015-08-25 12:46 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150824075018.GB20106@gmail.com \
--to=mingo@kernel.org \
--cc=dave@sr71.net \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux@horizon.com \
--cc=linux@rasmusvillemoes.dk \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=rientjes@google.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.