From: Andi Kleen <andi@firstfloor.org>
To: linux-kernel@vger.kernel.org
Cc: x86@kernel.org, mingo@kernel.org, torvalds@linux-foundation.org,
Andi Kleen <ak@linux.intel.com>
Subject: [PATCH 09/13] x86: Move cond_resched into the out of line get_user code
Date: Fri, 9 Aug 2013 16:04:16 -0700 [thread overview]
Message-ID: <1376089460-5459-10-git-send-email-andi@firstfloor.org> (raw)
In-Reply-To: <1376089460-5459-1-git-send-email-andi@firstfloor.org>
From: Andi Kleen <ak@linux.intel.com>
CONFIG_PREEMPT_VOLUNTARY kernels always do a cond_resched in get_user().
Currently this is done in the caller and in a inefficient way (multiple
function calls to decide to do nothing). Move the reschedule check
into the low level functions instead, where it can be merged cheaply
with the address limit check.
For the DEBUG_SLEEP case we still do the call in the caller.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
arch/x86/include/asm/uaccess.h | 3 +--
arch/x86/lib/getuser.S | 9 +++++----
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 6cbe976..1ab2d2a 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -165,7 +165,7 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL))
int __ret_gu; \
register __inttype(*(ptr)) __val_gu asm("%edx"); \
__chk_user_ptr(ptr); \
- might_fault(); \
+ might_fault_debug_only(); \
asm volatile("call __get_user_%P3" \
: "=a" (__ret_gu), "=r" (__val_gu) \
: "0" (ptr), "i" (sizeof(*(ptr)))); \
@@ -541,4 +541,3 @@ extern struct movsl_mask {
#endif
#endif /* _ASM_X86_UACCESS_H */
-
diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S
index a451235..0288aa9 100644
--- a/arch/x86/lib/getuser.S
+++ b/arch/x86/lib/getuser.S
@@ -33,11 +33,12 @@
#include <asm/thread_info.h>
#include <asm/asm.h>
#include <asm/smap.h>
+#include "user-common.h"
.text
ENTRY(__get_user_1)
CFI_STARTPROC
- GET_THREAD_INFO(%_ASM_DX)
+ GET_THREAD_AND_SCHEDULE %_ASM_DX
cmp TI_addr_limit(%_ASM_DX),%_ASM_AX
jae bad_get_user
ASM_STAC
@@ -52,7 +53,7 @@ ENTRY(__get_user_2)
CFI_STARTPROC
add $1,%_ASM_AX
jc bad_get_user
- GET_THREAD_INFO(%_ASM_DX)
+ GET_THREAD_AND_SCHEDULE %_ASM_DX
cmp TI_addr_limit(%_ASM_DX),%_ASM_AX
jae bad_get_user
ASM_STAC
@@ -67,7 +68,7 @@ ENTRY(__get_user_4)
CFI_STARTPROC
add $3,%_ASM_AX
jc bad_get_user
- GET_THREAD_INFO(%_ASM_DX)
+ GET_THREAD_AND_SCHEDULE %_ASM_DX
cmp TI_addr_limit(%_ASM_DX),%_ASM_AX
jae bad_get_user
ASM_STAC
@@ -83,7 +84,7 @@ ENTRY(__get_user_8)
#ifdef CONFIG_X86_64
add $7,%_ASM_AX
jc bad_get_user
- GET_THREAD_INFO(%_ASM_DX)
+ GET_THREAD_AND_SCHEDULE %_ASM_DX
cmp TI_addr_limit(%_ASM_DX),%_ASM_AX
jae bad_get_user
ASM_STAC
--
1.8.3.1
next prev parent reply other threads:[~2013-08-09 23:07 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-09 23:04 Re-tune x86 uaccess code for PREEMPT_VOLUNTARY Andi Kleen
2013-08-09 23:04 ` [PATCH 01/13] x86: Add 1/2/4/8 byte optimization to 64bit __copy_{from,to}_user_inatomic Andi Kleen
2013-08-09 23:04 ` [PATCH 02/13] x86: Include linux/sched.h in asm/uaccess.h Andi Kleen
2013-08-09 23:04 ` [PATCH 03/13] tree-sweep: Include linux/sched.h for might_sleep users Andi Kleen
2013-08-09 23:04 ` [PATCH 04/13] Move might_sleep and friends from kernel.h to sched.h Andi Kleen
2013-08-09 23:04 ` [PATCH 05/13] sched: mark should_resched() __always_inline Andi Kleen
2013-08-09 23:04 ` [PATCH 06/13] x86: Add 32bit versions of SAVE_ALL/RESTORE_ALL to calling.h Andi Kleen
2013-08-09 23:04 ` [PATCH 07/13] Add might_fault_debug_only() Andi Kleen
2013-08-14 18:24 ` Michael S. Tsirkin
2013-08-09 23:04 ` [PATCH 08/13] x86: Move cond_resched into the out of line put_user code Andi Kleen
2013-08-09 23:04 ` Andi Kleen [this message]
2013-08-09 23:04 ` [PATCH 10/13] x86: Move cond resched for copy_{from,to}_user into low level code 64bit Andi Kleen
2013-08-10 15:42 ` Linus Torvalds
2013-08-10 16:10 ` Andi Kleen
2013-08-10 16:27 ` Linus Torvalds
2013-08-10 18:23 ` Borislav Petkov
2013-08-10 19:05 ` Jörn Engel
2013-08-20 21:03 ` KOSAKI Motohiro
2013-08-15 5:04 ` Michael S. Tsirkin
2013-08-09 23:04 ` [PATCH 11/13] sched: Inline the need_resched test into the caller for _cond_resched Andi Kleen
2013-08-09 23:04 ` [PATCH 12/13] x86: move __copy_*_nocache might fault check out of line Andi Kleen
2013-08-09 23:04 ` [PATCH 13/13] x86: drop cond rescheds from __copy_{from,to}_user Andi Kleen
2013-08-10 4:42 ` Re-tune x86 uaccess code for PREEMPT_VOLUNTARY H. Peter Anvin
2013-08-10 5:55 ` Mike Galbraith
2013-08-10 16:09 ` H. Peter Anvin
2013-08-10 16:43 ` Linus Torvalds
2013-08-10 17:18 ` H. Peter Anvin
2013-08-10 18:51 ` Linus Torvalds
2013-08-10 19:18 ` H. Peter Anvin
2013-08-10 20:26 ` H. Peter Anvin
2013-08-10 23:00 ` H. Peter Anvin
2013-08-11 4:17 ` Mike Galbraith
2013-08-11 4:27 ` H. Peter Anvin
2013-08-11 4:36 ` Mike Galbraith
2013-08-11 4:57 ` H. Peter Anvin
2013-08-11 5:58 ` Mike Galbraith
2013-08-13 18:09 ` H. Peter Anvin
2013-08-13 18:12 ` Andi Kleen
2013-08-14 18:27 ` Michael S. Tsirkin
2013-08-14 22:08 ` Andi Kleen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1376089460-5459-10-git-send-email-andi@firstfloor.org \
--to=andi@firstfloor.org \
--cc=ak@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).