From: Andi Kleen <andi@firstfloor.org>
To: peterz@infradead.org
Cc: x86@kernel.org, linux-kernel@vger.kernel.org,
Andi Kleen <ak@linux.intel.com>
Subject: [PATCH 2/4] x86: Move copy_from_user_nmi() inline
Date: Fri, 10 Oct 2014 16:25:15 -0700 [thread overview]
Message-ID: <1412983517-12419-3-git-send-email-andi@firstfloor.org> (raw)
In-Reply-To: <1412983517-12419-1-git-send-email-andi@firstfloor.org>
From: Andi Kleen <ak@linux.intel.com>
Move copy_from_user_nmi() inline. This allows the compiler to directly
do the __builtin_constant_p() optimizations in __copy_from_user_nocheck.
This then allows to optimize an 8 byte (32bit) or 16byte copy (64bit)
into two direct __get_user() instead of using the generic copy function.
This covers the 8/16 byte copies dump_stack uses when called from
the performance critical perf nmi pmi handler.
First this is much faster by itself (single memory access vs complicated
function). But it also is a lot faster for any page fault, which are
common in backtracing. Currently copy_from_user() does every page
fault twice, to generate an exact unread-bytes count. This adds a lot
of overhead. The inline __get_user code can do this without significant
overhead, it bails out on the first fault.
copy_from_user_nmi() is only placed from a few places, so there
isn't any significant code size increase from inlining this.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
arch/x86/include/asm/uaccess.h | 29 +++++++++++++++++++++++++++--
arch/x86/lib/usercopy.c | 36 ------------------------------------
2 files changed, 27 insertions(+), 38 deletions(-)
delete mode 100644 arch/x86/lib/usercopy.c
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index e50a84f..30c391c 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -523,8 +523,6 @@ struct __large_struct { unsigned long buf[100]; };
#define put_user_ex(x, ptr) \
__put_user_size_ex((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)))
-extern unsigned long
-copy_from_user_nmi(void *to, const void __user *from, unsigned long n);
extern __must_check long
strncpy_from_user(char *dst, const char __user *src, long count);
@@ -741,5 +739,32 @@ copy_to_user(void __user *to, const void *from, unsigned long n)
#undef __copy_from_user_overflow
#undef __copy_to_user_overflow
+/*
+ * We rely on the nested NMI work to allow atomic faults from the NMI path; the
+ * nested NMI paths are careful to preserve CR2.
+ *
+ * Inline this function so that the caller gets the __builtin_constant_p
+ * optimizations in __copy_from_user_nocheck
+ */
+static __must_check __always_inline unsigned long
+copy_from_user_nmi(void *to, const void __user *from, unsigned long n)
+{
+ unsigned long ret;
+
+ if (__range_not_ok(from, n, user_addr_max()))
+ return 0;
+
+ /*
+ * Even though this function is typically called from NMI/IRQ context
+ * disable pagefaults so that its behaviour is consistent even when
+ * called form other contexts.
+ */
+ pagefault_disable();
+ ret = __copy_from_user_inatomic(to, from, n);
+ pagefault_enable();
+
+ return ret;
+}
+
#endif /* _ASM_X86_UACCESS_H */
diff --git a/arch/x86/lib/usercopy.c b/arch/x86/lib/usercopy.c
deleted file mode 100644
index ddf9ecb..0000000
--- a/arch/x86/lib/usercopy.c
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * User address space access functions.
- *
- * For licencing details see kernel-base/COPYING
- */
-
-#include <linux/highmem.h>
-#include <linux/module.h>
-
-#include <asm/word-at-a-time.h>
-#include <linux/sched.h>
-
-/*
- * We rely on the nested NMI work to allow atomic faults from the NMI path; the
- * nested NMI paths are careful to preserve CR2.
- */
-unsigned long
-copy_from_user_nmi(void *to, const void __user *from, unsigned long n)
-{
- unsigned long ret;
-
- if (__range_not_ok(from, n, TASK_SIZE))
- return 0;
-
- /*
- * Even though this function is typically called from NMI/IRQ context
- * disable pagefaults so that its behaviour is consistent even when
- * called form other contexts.
- */
- pagefault_disable();
- ret = __copy_from_user_inatomic(to, from, n);
- pagefault_enable();
-
- return ret;
-}
-EXPORT_SYMBOL_GPL(copy_from_user_nmi);
--
1.9.3
next prev parent reply other threads:[~2014-10-10 23:25 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-10 23:25 Updated perf backtrace improvement patchkit Andi Kleen
2014-10-10 23:25 ` [PATCH 1/4] Move pagefault_enable/disable to own include file Andi Kleen
2014-10-10 23:25 ` Andi Kleen [this message]
2014-10-10 23:25 ` [PATCH 3/4] x86: Optimize enhanced copy user fault handling Andi Kleen
2014-10-10 23:25 ` [PATCH 4/4] x86: Use the page tables to look up kernel addresses in backtrace Andi Kleen
2014-10-11 0:24 ` Chuck Ebbert
2014-10-11 0:33 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1412983517-12419-3-git-send-email-andi@firstfloor.org \
--to=andi@firstfloor.org \
--cc=ak@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.