From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758923AbbJINQo (ORCPT ); Fri, 9 Oct 2015 09:16:44 -0400 Received: from terminus.zytor.com ([198.137.202.10]:46296 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753920AbbJINQl (ORCPT ); Fri, 9 Oct 2015 09:16:41 -0400 Date: Fri, 9 Oct 2015 06:15:42 -0700 From: tip-bot for Andy Lutomirski Message-ID: Cc: bp@alien8.de, peterz@infradead.org, brgerst@gmail.com, luto@amacapital.net, mingo@kernel.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, luto@kernel.org, tglx@linutronix.de, hpa@zytor.com, dvlasenk@redhat.com Reply-To: dvlasenk@redhat.com, linux-kernel@vger.kernel.org, torvalds@linux-foundation.org, luto@kernel.org, tglx@linutronix.de, hpa@zytor.com, brgerst@gmail.com, mingo@kernel.org, luto@amacapital.net, bp@alien8.de, peterz@infradead.org In-Reply-To: <9fc53eda4a5b924070952f12fa4ae3e477640a07.1444091585.git.luto@kernel.org> References: <9fc53eda4a5b924070952f12fa4ae3e477640a07.1444091585.git.luto@kernel.org> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/asm] x86/entry: Split and inline prepare_exit_to_usermode() Git-Commit-ID: 39b48e575e92e31251b74b4b48cea2129cee90bd X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 39b48e575e92e31251b74b4b48cea2129cee90bd Gitweb: http://git.kernel.org/tip/39b48e575e92e31251b74b4b48cea2129cee90bd Author: Andy Lutomirski AuthorDate: Mon, 5 Oct 2015 17:48:23 -0700 Committer: Ingo Molnar CommitDate: Fri, 9 Oct 2015 09:41:13 +0200 x86/entry: Split and inline prepare_exit_to_usermode() GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part. Help it out with prepare_exit_to_usermode() by splitting it into two parts and inline the hot part. Saves 6-8 cycles for compat syscalls. Signed-off-by: Andy Lutomirski Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Link: http://lkml.kernel.org/r/9fc53eda4a5b924070952f12fa4ae3e477640a07.1444091585.git.luto@kernel.org Signed-off-by: Ingo Molnar --- arch/x86/entry/common.c | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index d087421..66ccbd6 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -218,14 +218,12 @@ long syscall_trace_enter(struct pt_regs *regs) return syscall_trace_enter_phase2(regs, arch, phase1_result); } -/* Called with IRQs disabled. */ -__visible void prepare_exit_to_usermode(struct pt_regs *regs) -{ - if (IS_ENABLED(CONFIG_PROVE_LOCKING) && WARN_ON(!irqs_disabled())) - local_irq_disable(); - - lockdep_sys_exit(); +#define EXIT_TO_USERMODE_LOOP_FLAGS \ + (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ + _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY) +static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags) +{ /* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY, @@ -235,14 +233,6 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs) * work to clear some of the flags can sleep. */ while (true) { - u32 cached_flags = - READ_ONCE(pt_regs_to_thread_info(regs)->flags); - - if (!(cached_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | - _TIF_UPROBE | _TIF_NEED_RESCHED | - _TIF_USER_RETURN_NOTIFY))) - break; - /* We have work to do. */ local_irq_enable(); @@ -266,7 +256,30 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs) /* Disable IRQs and retry */ local_irq_disable(); + + cached_flags = READ_ONCE(pt_regs_to_thread_info(regs)->flags); + + if (!(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) + break; + } +} + +/* Called with IRQs disabled. */ +__visible inline void prepare_exit_to_usermode(struct pt_regs *regs) +{ + u32 cached_flags; + + if (IS_ENABLED(CONFIG_PROVE_LOCKING) && WARN_ON(!irqs_disabled())) + local_irq_disable(); + + lockdep_sys_exit(); + + cached_flags = + READ_ONCE(pt_regs_to_thread_info(regs)->flags); + + if (unlikely(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) + exit_to_usermode_loop(regs, cached_flags); user_enter(); }